Skip to content
AI Detection & Response (...
/Deployments/
/
AWS

Deploy AIDR Container on AWS

These installation steps show you how to install locally into a Minikube cluster. Instructions are also valid for Kubernetes provided by Docker Desktop and a full-fledged Kubernetes cluster. Running locally with Minikube introduces extra latency due to the prompt injection classification model. Latency is not present in a full-fledged Kubernetes cluster.

Before deploying Model Scanner, make sure you understand:

Select OS

Select your operating system to view installation instructions.

  1. Open a terminal and create an environment variable file with Quay credentials. Use the following examples. Replace %QUAY_USERNAME% and %QUAY_PASSWORD% with actual values.

    cat << EOF > env
HIDDENLAYER_USERNAME=%QUAY_USERNAME%
HIDDENLAYER_PASSWORD=%QUAY_PASSWORD%
EOF
    Special Characters

    Special characters will require wrapping the value in a string.

  2. Load the environment variables into your shell.

    source ./env
Select Deployment Type

Select your deployment type to view instructions.

A Hybrid deployment sends metadata to the HiddenLayer AISec Platform. See Hybrid and Disconnected Deployments for information about each deployment type.

  1. Create a file named .env.local.

    • Use the comments in the example to find configurations for your deployment needs.

    • File changes to note:

      • For the EU region, uncomment hl_region=eu.
      • Replace <license> with your AIDR license.
      • Replace <client id> and <client secret> with your HiddenLayer API key and secret.
      • Replace <aws_access_key> and <aws_access_secret> with information for your AWS instance.
    cat << EOF > .env.local
namespace:
  name=aidr-genai

image:
  ## By default, the image below is deployed; this image is for CPU deployments
  repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai
  ## To set up a GPU deployment, uncomment the line below to use the CUDA image
  # repository=quay.io/hiddenlayer/distro-enterprise-aidr-genai-cuda
  ## By default, the latest version is installed
  ## Uncomment the following to specify a specific version
  # tag=25.5.2

## By default, hybrid connection will connect to the 'US' region
## Uncomment the following option to change to a different supported region  
# hl_region=eu

resources:
  ## For CPU deployments, 8 vCPUs are recommended
  ## For GPU deployments, 4 vCPUs are recommended
  requests:
    cpu=8
  ## For GPU deployments, uncomment this to allocate 1 GPU per instance
  # limits:
    # nvidia.com/gpu=1

## Set number of instances for deployment
replicas:
  min=1
  max=1

## Specify a specific node group if necessary
## For instance, GPU deployments may require specific nodes to access a GPU
# nodeGroup=<nodegroup_name>

config:
  HL_LICENSE=<license>
  HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<aws_access_key>
  HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<aws_access_secret>

  ## This must match the number of vCPUs assigned above
  OMP_NUM_THREADS=8

## By default, the hybrid connection type is used and requires a client id and secret
  HL_LLM_PROXY_CLIENT_ID=<client_id>
  HL_LLM_PROXY_CLIENT_SECRET=<client_secret>

  ## Uncomment the following to disable the connection to the AISec Platform
  ## in this mode, the client id and secret are not required
  # HL_LLM_PROXY_MLDR_CONNECTION_TYPE=disabled
EOF
Policy Configuration Environment Variables

For other policy configuration environment variables, see AIDR Configuration.

  1. Make sure Docker Desktop is running.

  2. Run the following command to login to the HiddenLayer Quay repository.

    docker login --username=$HIDDENLAYER_USERNAME --password=$HIDDENLAYER_PASSWORD quay.io

  3. Run the AIDR container.

    docker run --platform linux/amd64 --env-file  .env.local -p 8000:8000 quay.io/hiddenlayer/distro-enterprise-aidr-genai:latest

Health Check

To check that the proxy is up and running, open a new terminal and use the following command to ping the health monitor.

curl http://localhost:8000/health

Generate Manifest

To write the generated manifest to a file (manifest.yml), open a terminal and run the following command.

  • You must run the AI Detection & Response installer before generating a manifest.
  • Change latest to the AI Detection & Response version that you use.
docker run --env-file ./env -v pwd/config:/src/config quay.io/hiddenlayer/distro-enterprise-aidr-genai-installer:latest > manifest.yml
Previous page
Next page