LLM Connection Configuration - HiddenLayer Docs

While there are circumstances in which AI Runtime Security can be operated using an open, freely accessible LLM, typically organizations are using LLMs protected by API keys, running on dedicated company-hosted instances, and otherwise requiring container-level configuration to interact seamlessly with Runtime Security. It’s worth noting that operating Runtime Security in forward-proxy (“enriched”) mode requires the LLM to be configured in the container settings, and not via API key. Additionally, configuring the connection in the container spares the necessity of sending the key with every request. The container environment variables in this section are used to configure a backend connection to LLMs on the container level. Note that many of them do not have defaults, as they are only accessed when connecting to a specific LLM instance.

AWS Configuration

Runtime Security can be configured to route traffic to a single AWS account or multiple AWS accounts.

If there is only one AWS account configured, this account is used as the default for routing all traffic.
If multiple AWS accounts are registered, while there is a default account, it is not used for routing all traffic. Requests with incorrect AWS account information will result in an error message.
- There must be a default registered account. A single account configuration will be set as the default account.
- For any additional named credential sets, the suffix X in the environment variables should be the actual AWS Access Key ID. See the Single AWS Account and Multiple AWS Accounts examples below.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT`	True		Default AWS Access Key ID
`HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT`	True		Default AWS Secret Access Key
`HL_LLM_PROXY_AWS_SESSION_TOKEN_DEFAULT`	False		Default AWS Session Token (required if using temporary credentials)
`HL_LLM_PROXY_AWS_REGION_DEFAULT`	False	`us-east-1`	Default AWS Region used if none is specified for another set
`HL_LLM_PROXY_AWS_BEDROCK_BASE_URL`	False	`https://bedrock-runtime.{region}.amazonaws.com`	Default Bedrock base URL used if none is specified for another set
`HL_LLM_PROXY_AWS_SAGEMAKER_BASE_URL`	False	`https://runtime.sagemaker.{region}.amazonaws.com`	Default Sagemaker base URL used if none is specified for another set
`HL_LLM_PROXY_AWS_ACCESS_KEY_ID_X`	False		AWS Secret Access Key for the credential set
`HL_LLM_PROXY_AWS_SESSION_TOKEN_X`	False		AWS Session Token for the credential set
`HL_LLM_PROXY_AWS_REGION_X`	False	`Uses default region`	AWS Region for the credential set
`HL_LLM_PROXY_AWS_BEDROCK_BASE_URL_X`	False	`Uses default Bedrock`	Bedrock base URL for the credential set
`HL_LLM_PROXY_AWS_SAGEMAKER_BASE_URL_X`	False	`Uses default Sagemaker`	Sagemaker base URL for the credential set

Default AWS Account

If only one AWS account is registered with Runtime Security, this account is used by default to route traffic. Example - Default AWS Account This will create a default credential set. The application can then use this credential set to configure AWS clients.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1

Additional AWS Accounts

Runtime Security can be configured to route traffic to multiple AWS accounts. If there are multiple keys registered, there is no default account for routing traffic. Requests with incorrect AWS account information will result in an error message. When configuring multiple AWS accounts, you must create a default account and additional accounts. Example - Default AWS Account This will create a default credential set.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1

Example - Additional AWS Accounts If your AWS Access Key ID is AKIAXXXXXXXXFOO, this will create a credential set named AKIAXXXXXXXXFOO.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SESSION_TOKEN_AKIAXXXXXXXXFOO=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_AKIAXXXXXXXXFOO=us-east-1

AWS Credential Sets Environment Variables

There must be a default AWS account, for either a single registered account or multiple registered accounts. For any additional named credential sets, the suffix X in the environment variables should be the actual AWS Access Key ID. For example, if your AWS Access Key ID is AKIAXXXXXXXXFOO, set HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO and corresponding variables using that exact key ID. This creates a credential set named after that AWS Access Key ID.

AWS Secrets Manager

Arbitrary settings can be sourced from the AWS Secrets Manager. This is intended for Instance Profile authentication use only. The environment variable is HL_LLM_PROXY_SECRETS_<MANAGER NAME>_SECRET_<PROXY SETTING NAME>.

For <MANAGER NAME>, use AWS.
For <PROXY SETTING NAME>, use any of the environment variable names for a setting.

The following examples will pull the HL_LICENSE and HL_LLM_PROXY_CLIENT_ID from a secret named HiddenLayer_Runtime in the AWS Secrets Manager.

HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LICENSE=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_Runtime-abCdEF
HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LLM_PROXY_CLIENT_ID=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_Runtime-abCdEF

When creating new secrets, AWS encourages you to store the secrets in a JSON blob. For example, '{"HL_LLM_PROXY_CLIENT_ID": "xxxx-xxxx-xxxx-xxxx"}'.

This implementation requires the keys in this JSON blob to match the Proxy env var setting name.
Alternatively, you can store a plaintext value for the secret.

Azure Configuration

Runtime Security can route traffic to Azure OpenAI and Azure ML. Authenticate with a Microsoft Entra ID app registration (service principal) using the Azure settings, or with an Azure OpenAI API key using the Azure OpenAI settings.

Azure

Set these variables when the proxy authenticates to Azure with a Microsoft Entra ID app registration.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_AZURE_TENANT_ID`	False		Microsoft Entra ID tenant ID for the app registration used to authenticate to Azure
`HL_LLM_PROXY_AZURE_CLIENT_ID`	False		Application (client) ID of the Microsoft Entra ID app registration used to authenticate to Azure
`HL_LLM_PROXY_AZURE_CLIENT_SECRET`	False		Client secret of the Microsoft Entra ID app registration used to authenticate to Azure
`HL_LLM_PROXY_AZURE_BASE_URL`	False	`https://<resource-name>.openai.azure.com`	Base URL of the Azure resource the proxy routes requests to, such as an Azure OpenAI or Azure ML endpoint
`HL_LLM_PROXY_AZURE_REGION`	False	`eastus`	Azure region of the target resource. Used when the region is not already part of the base URL

Azure OpenAI

Set these variables when the proxy authenticates to Azure OpenAI with an API key.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_OPENAI_AZURE_API_KEY`	False		API key for the Azure OpenAI resource. The proxy uses this key to authenticate requests to Azure OpenAI
`HL_LLM_PROXY_OPENAI_AZURE_SCHEME`	False	`https`	URL scheme used to construct the Azure OpenAI endpoint. Use `https` unless connecting to a non-TLS endpoint
`HL_LLM_PROXY_OPENAI_AZURE_HOST`	False	`<resource-name>.openai.azure.com`	Hostname of the Azure OpenAI resource, without the scheme

OpenAI Configuration

Runtime Security can be configured to route traffic to an OpenAI account.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_OPENAI_DEFAULT_MODEL`	False		Enforces the use of only the model specified in this setting. For example, `gpt-4o`.
`HL_LLM_PROXY_OPENAI_BASE_URL`	False	`https://api.openai.com`	Default OpenAI base URL used if none is specified for another set
`HL_LLM_PROXY_OPENAI_API_KEY`	False		The API key to access OpenAI

Hugging Face Configuration

Runtime Security can be configured to route traffic to a Hugging Face model.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_HUGGINGFACE_TGI_PROVIDER`	False	`huggingface-tgi`	Identifying Hugging Face as the model provider
`HL_LLM_PROXY_HUGGINGFACE_TGI_NAMES`	False	`Qwen/Qwen3-0.6B`	The Hugging Face model name
`HL_LLM_PROXY_HUGGINGFACE_TGI_BASE_URLS`	False		Default Hugging Face base URL used if none is specified for another set

Custom Configuration

Runtime Security can be configured to route traffic to a custom model, like Ollama.

Environment Key	Required	Example Value	Description
`HL_LLM_PROXY_CUSTOM_{{MODELNAME}}`	False	`HL_LLM_PROXY_CUSTOM_LLAMA=tinyllama`	Used to provide an appropriate name for the model
`HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_PROVIDER`	False	`HL_LLM_PROXY_CUSTOM_LLAMA_PROVIDER=ollama`	Provider of the custom model
`HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_BASE_URL`	False	`HL_LLM_PROXY_CUSTOM_LLAMA_BASE_URL=http://ollama:11434`	Base URL of the endpoint where the model can be queried
`HL_LLM_PROXY_CUSTOM_{{MODELNAME}}_API_KEY`	False		The API key for the custom model.

​AWS Configuration

​Default AWS Account

​Additional AWS Accounts

​AWS Credential Sets Environment Variables

​AWS Secrets Manager

​Azure Configuration

​Azure

​Azure OpenAI

​OpenAI Configuration

​Hugging Face Configuration

​Custom Configuration