> ## Documentation Index
> Fetch the complete documentation index at: https://docs.hiddenlayer.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Connection Configuration

While there are circumstances in which AI Runtime Security can be operated using an open, freely accessible LLM, typically organizations are using LLMs protected by API keys, running on dedicated company-hosted instances, and otherwise requiring container-level configuration to interact seamlessly with Runtime Security. It’s worth noting that operating Runtime Security in forward-proxy (“enriched”) mode requires the LLM to be configured in the container settings, and not via API key. Additionally, configuring the connection in the container spares the necessity of sending the key with every request.

The container environment variables in this section are used to configure a backend connection to LLMs on the container level. Note that many of them do not have defaults, as they are only accessed when connecting to a specific LLM instance.

## AWS Configuration

Runtime Security can be configured to route traffic to a single AWS account or multiple AWS accounts.

* If there is only one AWS account configured, this account is used as the default for routing all traffic.
* If multiple AWS accounts are registered, while there is a default account, it is not used for routing all traffic. Requests with incorrect AWS account information will result in an error message.

  * There must be a default registered account. A single account configuration will be set as the default account.
  * For any additional named credential sets, the suffix X in the environment variables should be the actual AWS Access Key ID. See the [Single AWS Account](#single-aws-account) and [Multiple AWS Accounts](#multiple-aws-accounts) examples below.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT` | True | | Default AWS Access Key ID |
| `HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT` | True | | Default AWS Secret Access Key |
| `HL_LLM_PROXY_AWS_SESSION_TOKEN_DEFAULT` | False | | Default AWS Session Token (required if using temporary credentials) |
| `HL_LLM_PROXY_AWS_REGION_DEFAULT` | False | `us-east-1` | Default AWS Region used if none is specified for another set |
| `HL_LLM_PROXY_AWS_BEDROCK_BASE_URL` | False | `https://bedrock-runtime.{region}.amazonaws.com` | Default Bedrock base URL used if none is specified for another set |
| `HL_LLM_PROXY_AWS_SAGEMAKER_BASE_URL` | False | `https://runtime.sagemaker.{region}.amazonaws.com` | Default Sagemaker base URL used if none is specified for another set |
| `HL_LLM_PROXY_AWS_ACCESS_KEY_ID_X` | False | | AWS Secret Access Key for the credential set |
| `HL_LLM_PROXY_AWS_SESSION_TOKEN_X` | False | | AWS Session Token for the credential set |
| `HL_LLM_PROXY_AWS_REGION_X` | False | `Uses default region` | AWS Region for the credential set |
| `HL_LLM_PROXY_AWS_BEDROCK_BASE_URL_X` | False | `Uses default Bedrock` | Bedrock base URL for the credential set |
| `HL_LLM_PROXY_AWS_SAGEMAKER_BASE_URL_X` | False | `Uses default Sagemaker` | Sagemaker base URL for the credential set |

### Default AWS Account

If only one AWS account is registered with Runtime Security, this account is used by default to route traffic.

**Example - Default AWS Account**

This will create a default credential set. The application can then use this credential set to configure AWS clients.

```
export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1
```

### Additional AWS Accounts

Runtime Security can be configured to route traffic to multiple AWS accounts. If there are multiple keys registered, there is no default account for routing traffic. Requests with incorrect AWS account information will result in an error message.

When configuring multiple AWS accounts, you must create a default account and additional accounts.

**Example - Default AWS Account**

This will create a default credential set.

```
export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1
```

**Example - Additional AWS Accounts**

If your AWS Access Key ID is `AKIAXXXXXXXXFOO`, this will create a credential set named `AKIAXXXXXXXXFOO`.

```
export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SESSION_TOKEN_AKIAXXXXXXXXFOO=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_AKIAXXXXXXXXFOO=us-east-1
```

### AWS Credential Sets Environment Variables

There must be a default AWS account, for either a single registered account or multiple registered accounts.

For any additional named credential sets, the suffix `X` in the environment variables should be **the actual AWS Access Key ID**. For example, if your AWS Access Key ID is `AKIAXXXXXXXXFOO`, set `HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO` and corresponding variables using that exact key ID. This creates a credential set named after that AWS Access Key ID.

### AWS Secrets Manager

Arbitrary settings can be sourced from the AWS Secrets Manager. This is intended for Instance Profile authentication use only.

The environment variable is `HL_LLM_PROXY_SECRETS_<MANAGER NAME>_SECRET_<PROXY SETTING NAME>`.

* For `<MANAGER NAME>`, use AWS.
* For `<PROXY SETTING NAME>`, use any of the environment variable names for a setting.

The following examples will pull the `HL_LICENSE` and `HL_LLM_PROXY_CLIENT_ID` from a secret named `HiddenLayer_Runtime` in the AWS Secrets Manager.

```
HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LICENSE=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_Runtime-abCdEF
HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LLM_PROXY_CLIENT_ID=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_Runtime-abCdEF
```

When creating new secrets, AWS encourages you to store the secrets in a JSON blob. For example, `'{"HL_LLM_PROXY_CLIENT_ID": "xxxx-xxxx-xxxx-xxxx"}'`.

* This implementation requires the keys in this JSON blob to match the Proxy env var setting name.
* Alternatively, you can store a plaintext value for the secret.

## Azure Configuration

Runtime Security can route traffic to Azure OpenAI and Azure ML. Authenticate with a Microsoft Entra ID app registration (service principal) using the Azure settings, or with an Azure OpenAI API key using the Azure OpenAI settings.

### Azure

Set these variables when the proxy authenticates to Azure with a Microsoft Entra ID app registration.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_AZURE_TENANT_ID` | False | | Microsoft Entra ID tenant ID for the app registration used to authenticate to Azure |
| `HL_LLM_PROXY_AZURE_CLIENT_ID` | False | | Application (client) ID of the Microsoft Entra ID app registration used to authenticate to Azure |
| `HL_LLM_PROXY_AZURE_CLIENT_SECRET` | False | | Client secret of the Microsoft Entra ID app registration used to authenticate to Azure |
| `HL_LLM_PROXY_AZURE_BASE_URL` | False | `https://<resource-name>.openai.azure.com` | Base URL of the Azure resource the proxy routes requests to, such as an Azure OpenAI or Azure ML endpoint |
| `HL_LLM_PROXY_AZURE_REGION` | False | `eastus` | Azure region of the target resource. Used when the region is not already part of the base URL |

### Azure OpenAI

Set these variables when the proxy authenticates to Azure OpenAI with an API key.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_OPENAI_AZURE_API_KEY` | False | | API key for the Azure OpenAI resource. The proxy uses this key to authenticate requests to Azure OpenAI |
| `HL_LLM_PROXY_OPENAI_AZURE_SCHEME` | False | `https` | URL scheme used to construct the Azure OpenAI endpoint. Use `https` unless connecting to a non-TLS endpoint |
| `HL_LLM_PROXY_OPENAI_AZURE_HOST` | False | `<resource-name>.openai.azure.com` | Hostname of the Azure OpenAI resource, without the scheme |

## OpenAI Configuration

Runtime Security can be configured to route traffic to an OpenAI account.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_OPENAI_DEFAULT_MODEL` | False | | Enforces the use of only the model specified in this setting. For example, `gpt-4o`. |
| `HL_LLM_PROXY_OPENAI_BASE_URL` | False | `https://api.openai.com` | Default OpenAI base URL used if none is specified for another set |
| `HL_LLM_PROXY_OPENAI_API_KEY` | False | | The API key to access OpenAI |

## Hugging Face Configuration

Runtime Security can be configured to route traffic to a Hugging Face model.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_HUGGINGFACE_TGI_PROVIDER` | False | `huggingface-tgi` | Identifying Hugging Face as the model provider |
| `HL_LLM_PROXY_HUGGINGFACE_TGI_NAMES` | False | `Qwen/Qwen3-0.6B` | The Hugging Face model name |
| `HL_LLM_PROXY_HUGGINGFACE_TGI_BASE_URLS` | False | | Default Hugging Face base URL used if none is specified for another set |

## Custom Configuration

Runtime Security can be configured to route traffic to a custom model, like Ollama.

| Environment Key | Required | Example Value | Description |
| - | - | - | - |
| `HL_LLM_PROXY_CUSTOM_{{MODELNAME}}` | False | `HL_LLM_PROXY_CUSTOM_LLAMA=tinyllama` | Used to provide an appropriate name for the model |
| `HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_PROVIDER` | False | `HL_LLM_PROXY_CUSTOM_LLAMA_PROVIDER=ollama` | Provider of the custom model |
| `HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_BASE_URL` | False | `HL_LLM_PROXY_CUSTOM_LLAMA_BASE_URL=http://ollama:11434` | Base URL of the endpoint where the model can be queried |
| `HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_API_KEY` | False | | The API key for the custom model. |
