# Deploy AI Runtime Security using Helm Deploy AI Runtime Security to a Kubernetes cluster using Helm. ## Prerequisites - Access to a Kubernetes cluster - [kubectl](https://kubernetes.io/docs/tasks/tools/) — Kubernetes command-line tool - [Helm](https://helm.sh/docs/intro/install/) — Kubernetes package manager (v3+) - [Resource Requirements](/docs/products/runtime/resource_requirements) — license keys, tools, and scaling guidance - [Hybrid and Disconnected Modes](/docs/products/runtime/hybrid_disconnected) — connection mode details ## Create a Helm Values File Create a `values.yaml` file to override the chart's default settings. Application configuration goes inside `config.settings.yaml` as an embedded YAML string block. Secrets (auth credentials and license) go in the `env:` block. Handling Secrets Sensitive values in `env:` should use the `secret:` prefix followed by the **base64-encoded** value (e.g., the output of `echo -n "my-value" | base64`). The chart automatically creates a Kubernetes Secret and injects it via `secretKeyRef`, so the plaintext value never appears in the pod spec. Choose the resource type for your deployment. - CPU: Deploy on resources that utilize CPUs only. This is the default image. - GPU: Deploy on resources that utilize a CUDA-enabled GPU. CPU Choose the connection mode for your deployment. See [Hybrid and Disconnected Modes](/docs/products/runtime/hybrid_disconnected) for details on what data is sent in each mode. Hybrid In **Hybrid** mode, metadata per inference is sent to the HiddenLayer Console to power visualizations and alerting. This requires authentication credentials. By default, prompts and responses are also sent to the Console so you can review Interactions in context. To disable prompt collection, set `log-chat-context` to `false` under `aidr-genai.detector.engine`. ```yaml aidr_genai: image: repository: distro-enterprise-aidr-genai env: HL_LLM_PROXY_CLIENT_ID: "secret:" HL_LLM_PROXY_CLIENT_SECRET: "secret:" HL_LICENSE: "secret:" HL_LLM_PROXY_TOKEN_URL: "https://auth.hiddenlayer.ai" HL_LLM_PROXY_MLDR_CONNECTION_TYPE: "hybrid" HL_LLM_PROXY_MLDR_BASE_URL: "https://api.us.hiddenlayer.ai" # US region # HL_LLM_PROXY_MLDR_BASE_URL: "https://api.eu.hiddenlayer.ai" # EU region HL_LLM_PROXY_LOG_LEVEL: "info" HL_LLM_PROXY_MLDR_COLLECT_PROMPT: "true" ``` Disconnected In **Disconnected** mode, no data is sent back to HiddenLayer. Authentication credentials are not required. ```yaml aidr_genai: image: repository: distro-enterprise-aidr-genai env: HL_LICENSE: "secret:" HL_LLM_PROXY_MLDR_CONNECTION_TYPE: "disabled" ``` GPU Choose the connection mode for your deployment. See [Hybrid and Disconnected Modes](/docs/products/runtime/hybrid_disconnected) for details on what data is sent in each mode. Hybrid In **Hybrid** mode, metadata per inference is sent to the HiddenLayer Console to power visualizations and alerting. This requires authentication credentials. By default, prompts and responses are also sent to the Console so you can review Interactions in context. To disable prompt collection, set `log-chat-context` to `false` under `aidr-genai.detector.engine`. ```yaml aidr_genai: image: repository: distro-enterprise-aidr-genai-cuda resources: limits: nvidia.com/gpu: 1 env: HL_LLM_PROXY_CLIENT_ID: "secret:" HL_LLM_PROXY_CLIENT_SECRET: "secret:" HL_LICENSE: "secret:" HL_LLM_PROXY_TOKEN_URL: "https://auth.hiddenlayer.ai" HL_LLM_PROXY_MLDR_CONNECTION_TYPE: "hybrid" HL_LLM_PROXY_MLDR_BASE_URL: "https://api.us.hiddenlayer.ai" # US region # HL_LLM_PROXY_MLDR_BASE_URL: "https://api.eu.hiddenlayer.ai" # EU region HL_LLM_PROXY_LOG_LEVEL: "info" HL_LLM_PROXY_MLDR_COLLECT_PROMPT: "true" ``` Disconnected In **Disconnected** mode, no data is sent back to HiddenLayer. Authentication credentials are not required. ```yaml aidr_genai: image: repository: distro-enterprise-aidr-genai-cuda resources: limits: nvidia.com/gpu: 1 env: HL_LICENSE: "secret:" HL_LLM_PROXY_MLDR_CONNECTION_TYPE: "disabled" ``` ## Deployment ### Log In to the HiddenLayer Helm Registry 1. Run the following command in a terminal to log in to the HiddenLayer registry. - The `username` is your Registry username. - The `password` is your License ID. - For more information, see [Resource Requirements](/docs/products/runtime/resource_requirements). ``` helm registry login registry.hiddenlayer.ai --username --password ``` ### Using Your Own Repository For users that would like to use their own repository, see the steps below on how to pull the relevant images. Skip these steps if this is not applicable to you. details summary Download AIDR Images - Click to expand Run the following commands in the terminal to download the AIDR images. Docker Command Fails When using Docker commands, like `docker pull`, if you get a permission denied message, try using `sudo docker`, like `sudo docker pull`. Alternatively, add the user to the appropriate Docker group for Docker daemon permissions. Choose the resource type for your deployment. - CPU: Deploy on resources that utilize CPUs only. This is the default image. - GPU: Deploy on resources that utilize a CUDA-enabled GPU. CPU 1. Run the following command in a terminal to log in to the HiddenLayer image registry. - The username is the Registry Username. - The password is the License ID. ``` docker login images.hiddenlayer.ai --username --password ``` 2. Run each of the following commands to pull the images. ``` docker pull --platform linux/amd64 images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-sdk-image:v1.14.0 docker pull --platform linux/amd64 images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-license-enforcer:0.6.0 ``` 3. Tag the images to a private registry. Replace `%YOUR-REGISTRY%` with your private registry information. ``` docker tag images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-sdk-image:v1.14.0 %YOUR-REGISTRY%/replicated-sdk-image:v1.14.0 docker tag images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-license-enforcer:0.6.0 %YOUR-REGISTRY%/replicated-license-enforcer:0.6.0 ``` 4. Push the images to a private registry. Replace `%YOUR-REGISTRY%` with your private registry information. **Note**: Make sure you are logged in to your private registry before pushing the images. ``` docker push %YOUR-REGISTRY%/replicated-sdk-image:v1.14.0 docker push %YOUR-REGISTRY%/replicated-license-enforcer:0.6.0 ``` GPU 1. Run the following command in a terminal to log in to the HiddenLayer image registry. - The username is the Registry Username. - The password is the License ID. ``` docker login images.hiddenlayer.ai --username --password ``` 2. Run each of the following commands to pull the images. ``` docker pull --platform linux/amd64 images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-sdk-image:v1.14.0 docker pull --platform linux/amd64 images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-license-enforcer:0.6.0 ``` 3. Tag the images to a private registry. Replace `%YOUR-REGISTRY%` with your private registry information. ``` docker tag images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-sdk-image:v1.14.0 %YOUR-REGISTRY%/replicated-sdk-image:v1.14.0 docker tag images.hiddenlayer.ai/proxy/aidr-genai/ghcr.io/hiddenlayer-engineering/replicated-license-enforcer:0.6.0 %YOUR-REGISTRY%/replicated-license-enforcer:0.6.0 ``` 4. Push the images to a private registry. Replace `%YOUR-REGISTRY%` with your private registry information. **Note**: Make sure you are logged in to your private registry before pushing the images. ``` docker push %YOUR-REGISTRY%/replicated-sdk-image:v1.14.0 docker push %YOUR-REGISTRY%/replicated-license-enforcer:0.6.0 ``` ### Install 1. Create a `values.yaml` file to customize installation. - See [Create a Helm Values File](#create-a-helm-values-file) above. 2. Run the following command to deploy Runtime Security. ``` helm upgrade --install aidr-genai oci://registry.hiddenlayer.ai/aidr-genai/stable/aidr-genai \ --namespace aidr-genai --create-namespace \ -f values.yaml ``` ### Verify the Deployment 1. Check that all pods are running: ``` kubectl get pods -n aidr-genai ``` 2. Port-forward the service to your local machine: ``` kubectl port-forward svc/aidr-genai 8000:80 -n aidr-genai ``` 3. Verify the health endpoint: ``` curl http://localhost:8000/health ``` ## Using the Interactions Endpoint Once deployed, you can analyze LLM input and output by sending requests to the Interactions endpoint: ``` curl -X POST http://:8000/detection/v1/interactions \ -H "Content-Type: application/json" \ -d '{ "metadata": { "model": "gpt-5", "requester_id": "user-1234", "provider": "openai" }, "input": { "messages": [ { "role": "user", "content": "What is the largest moon of Jupiter?" } ] }, "output": { "messages": [ { "role": "assistant", "content": "The largest moon of Jupiter is Ganymede." } ] } }' ``` For SDK examples and the full response format, see [Getting Started with Interactions](/docs/products/runtime/interactions). ## Additional Configuration The following sections cover additional configuration beyond the baseline. The examples in [Create a Helm Values File](#create-a-helm-values-file) include working defaults for all of these — adjust as needed after verifying your deployment. ### Horizontal Autoscaling The chart creates a Horizontal Pod Autoscaler (HPA) when `resources.targetUtilization.cpu` and `resources.requests.cpu` are set. Use `replicas.min` and `replicas.max` to control scaling bounds. Allocate **8 CPU per replica** and set `OMP_NUM_THREADS: 8` to match. Scale replicas to fill node capacity — for example, 4 replicas on a 32-vCPU node. See [Resource Requirements](/docs/products/runtime/resource_requirements) for detailed scaling guidance. ```yaml aidr_genai: resources: targetUtilization: cpu: 75 requests: cpu: 8 memory: 4096Mi limits: memory: 4096Mi replicas: min: 2 max: 8 ``` ### Egress via HTTP/HTTPS Proxy In **Hybrid** mode, the AI Runtime Security pods make outbound connections to the HiddenLayer Console (`auth.hiddenlayer.ai` and the regional `api.*.hiddenlayer.ai` endpoint). If your cluster routes external traffic through a forward proxy, configure it by setting the standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` environment variables in the `aidr_genai.env` block of your `values.yaml`. This section does not apply to Disconnected mode, which sends no outbound traffic to HiddenLayer. No Authentication ```yaml aidr_genai: env: # ...existing credentials and settings... HTTP_PROXY: "http://proxy.internal.example.com:8080" HTTPS_PROXY: "http://proxy.internal.example.com:8080" # Bypass the proxy for in-cluster and internal traffic NO_PROXY: "localhost,127.0.0.1,.svc,.svc.cluster.local,kubernetes.default.svc," ``` With Authentication If your proxy requires authentication, include the credentials in the proxy URL (`http://:@host:port`). Any special characters in the username or password (for example, `@`, `:`, `#`, `/`) must be URL-encoded, otherwise the URL will be parsed incorrectly and authentication will fail — for example, a password of `p@ss#word` becomes `p%40ss%23word`. Because the URL contains a secret, use the `secret:` prefix with the **base64-encoded** URL so the chart stores it in a Kubernetes Secret instead of the pod spec. Use single quotes with `echo -n` so the shell does not expand characters such as `$` in the credentials (e.g., `echo -n 'http://user:p%40ss%23word@proxy.internal.example.com:8080' | base64`). ```yaml aidr_genai: env: # ...existing credentials and settings... HTTP_PROXY: "secret:" HTTPS_PROXY: "secret:" # Bypass the proxy for in-cluster and internal traffic NO_PROXY: "localhost,127.0.0.1,.svc,.svc.cluster.local,kubernetes.default.svc," ``` Set NO_PROXY carefully Include cluster-internal destinations in `NO_PROXY` so in-cluster traffic (the Kubernetes API, service-to-service calls, and any internally hosted LLM the proxy forwards to) bypasses the forward proxy. Add your cluster's Pod and Service CIDRs and the hostnames of any internal model endpoints. Proxies that intercept TLS If your forward proxy terminates and re-signs TLS (SSL inspection), the pods must trust the proxy's CA certificate. Populate `config.ca_bundle.crt` with the CA bundle and set `SSL_CERT_FILE: $(CONFIG_PATH)/ca_bundle.crt` in `aidr_genai.env`. ```yaml aidr_genai: config: ca_bundle.crt: | -----BEGIN CERTIFICATE----- -----END CERTIFICATE----- env: # ...existing credentials and settings... SSL_CERT_FILE: $(CONFIG_PATH)/ca_bundle.crt ``` ### Detection Policy Configure a detection policy using environment variables inside the `values.yaml` file. What the following example does: - Uses the AI Runtime Security image for a CPU-only resource. - Will send results to the HiddenLayer Console (US region). - For Prompt Injection: - Detect high severity prompt injection, but will not block it (alert-only). - For Personally Identifiable Information (PII): - Detect PII in input and output, but will not block it (alert-only). - Redact PII in input and output, replacing the redacted content with an entity label (example: `[PHONE_NUMBER]`). ```yaml aidr_genai: image: repository: distro-enterprise-aidr-genai # CPU image # repository: distro-enterprise-aidr-genai-cuda # GPU image # resources: # GPU resources # limits: # nvidia.com/gpu: 1 env: HL_LLM_PROXY_CLIENT_ID: "secret:" HL_LLM_PROXY_CLIENT_SECRET: "secret:" HL_LICENSE: "secret:" HL_LLM_PROXY_TOKEN_URL: "https://auth.hiddenlayer.ai" HL_LLM_PROXY_MLDR_CONNECTION_TYPE: "hybrid" HL_LLM_PROXY_MLDR_BASE_URL: "https://api.us.hiddenlayer.ai" # US region # HL_LLM_PROXY_MLDR_BASE_URL: "https://api.eu.hiddenlayer.ai" # EU region HL_LLM_PROXY_LOG_LEVEL: "info" HL_LLM_PROXY_MLDR_COLLECT_PROMPT: "true" # Prompt Injection HL_LLM_SKIP_PROMPT_INJECTION_DETECTION: "false" # detection is enabled HL_LLM_PROXY_CONVICTION_SEVERITY_PROMPT_INJECTION: "High" # high severity threshold for detections HL_LLM_BLOCK_PROMPT_INJECTION: "false" # alert only, do not block # PII Detection & Redaction HL_LLM_SKIP_INPUT_PII_DETECTION: "false" # input PII detection enabled HL_LLM_SKIP_OUTPUT_PII_DETECTION: "false" # output PII detection enabled HL_LLM_BLOCK_INPUT_PII: "false" # alert only, do not block input HL_LLM_BLOCK_OUTPUT_PII: "false" # alert only, do not block output HL_LLM_REDACT_INPUT_PII: "true" # redact PII in input HL_LLM_REDACT_OUTPUT_PII: "true" # redact PII in output HL_LLM_REDACT_TYPE: "ENTITY" # replace with entity label e.g. [PHONE_NUMBER] ```