LLM Connection Configuration

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

While there are circumstances in which AIDR can be operated using an open, freely accessible LLM, typically organizations are using LLMs protected by API keys, running on dedicated company-hosted instances, and otherwise requiring container-level configuration to interact seamlessly with AIDR for GenAI. It’s worth noting that operating the AIDR proxy in forward-proxy (“enriched”) mode requires the LLM to be configured in the container settings, and not via API key. Additionally, configuring the connection in the container spares the necessity of sending the key with every request.

The container environment variables in this section are used to configure a backend connection to LLMs on the container level. Note that many of them do not have defaults, as they are only accessed when connecting to a specific LLM instance.

AWS Configuration

AIDR can be configured to route traffic to a single AWS account or multiple AWS accounts.

• If there is only one AWS account configured, this account is used as the default for routing all traffic.

• If multiple AWS accounts are registered, while there is a default account, it is not used for routing all traffic. Requests with incorrect AWS account information will result in an error message.

  • There must be a default registered account. A single account configuration will be set as the default account.
  • For any additional named credential sets, the suffix X in the environment variables should be the actual AWS Access Key ID. See the Single AWS Account and Multiple AWS Accounts examples below.

Default AWS Account

If only one AWS account is registered with AIDR GenAI, this account is used by default to route traffic.

Example - Default AWS Account

This will create a default credential set. The application can then use this credential set to configure AWS clients.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1

Additional AWS Accounts

AIDR can be configured to route traffic to multiple AWS accounts. If there are multiple keys registered, there is no default account for routing traffic. Requests with incorrect AWS account information will result in an error message.

When configuring multiple AWS accounts, you must create a default account and additional accounts.

Example - Default AWS Account

This will create a default credential set.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_DEFAULT=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SECRET_ACCESS_KEY_DEFAULT=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_DEFAULT=us-east-1

Example - Additional AWS Accounts

If your AWS Access Key ID is AKIAXXXXXXXXFOO, this will create a credential set named AKIAXXXXXXXXFOO.

export HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO=<your-secret-access-key>
export HL_LLM_PROXY_AWS_SESSION_TOKEN_AKIAXXXXXXXXFOO=<your-session-token>
export HL_LLM_PROXY_AWS_REGION_AKIAXXXXXXXXFOO=us-east-1

AWS Credential Sets Environment Variables

There must be a default AWS account, for either a single registered account or multiple registered accounts.

For any additional named credential sets, the suffix X in the environment variables should be the actual AWS Access Key ID. For example, if your AWS Access Key ID is AKIAXXXXXXXXFOO, set HL_LLM_PROXY_AWS_ACCESS_KEY_ID_AKIAXXXXXXXXFOO and corresponding variables using that exact key ID. This creates a credential set named after that AWS Access Key ID.

AWS Secrets Manager

Arbitrary settings can be sourced from the AWS Secrets Manager. This is intended for Instance Profile authentication use only.

The environment variable is HL_LLM_PROXY_SECRETS_<MANAGER NAME>_SECRET_<PROXY SETTING NAME>.

• For <MANAGER NAME>, use AWS.
• For <PROXY SETTING NAME>, use any of the environment variable names for a setting.

The following examples will pull the HL_LICENSE and HL_LLM_PROXY_CLIENT_ID from a secret named HiddenLayer_AIDR in the AWS Secrets Manager.

HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LICENSE=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_AIDR-abCdEF
HL_LLM_PROXY_SECRETS_AWS_SECRET_HL_LLM_PROXY_CLIENT_ID=arn:aws:secretsmanager:us-west-2:123456789012:secret:HiddenLayer_AIDR-abCdEF

When creating new secrets, AWS encourages you to store the secrets in a JSON blob. For example, '{"HL_LLM_PROXY_CLIENT_ID": "xxxx-xxxx-xxxx-xxxx"}'.

• This implementation requires the keys in this JSON blob to match the Proxy env var setting name.
• Alternatively, you can store a plaintext value for the secret.

Azure Configuration

AIDR can be configured to route traffic to an Azure OpenAI account.

Azure

Azure OpenAI

OpenAI Configuration

AIDR can be configured to route traffic to an OpenAI account.

Hugging Face Configuration

AIDR can be configured to route traffic to a Hugging Face model.

Custom Configuration

AIDR can be configured to route traffic to a custom model, like Ollama.