Resource Requirements

Copy

• Copy for LLM

  Copy page as Markdown for LLMs
• View as Markdown

  Open this page as Markdown
• Open in ChatGPT

  Get insights from ChatGPT
• Open in Claude

  Get insights from Claude
• Connect to Cursor

  Install MCP server on Cursor
• Connect to VS Code

  Install MCP server on VS Code

License Keys

Hybrid Mode

The following licenses and keys are required for deploying AIDR in Hybrid mode. If your organization doesn't have a license, contact HiddenLayer for more information. See Hybrid and Disconnected Modes for information about Model Scanner Hybrid mode.

• AIDR License Key: HiddenLayer Support will provide you with a license key. This key is required to start the LLM proxy container, and it will not run without a valid key. The license can be set as an environment variable and the installer will not run without the license being set as a value.

• Credentials to download AIDR container: Credentials for the HiddenLayer container repository are required to download the appropriate images. These can also be obtained from HiddenLayer Support or from your HiddenLayer technical contact.

• API Client ID and Client Secret: HiddenLayer API Client ID and Client Secret to interact with the AISec Platform Console. Get these from the Console or your Console Admin.

  • API Permissions

    • Inferences: write
    • Model Inventory: write

  • Links to Console

    • Link to the US Console
    • Link to the EU Console

Disconnected Mode

The following licenses and keys are required for deploying AIDR in Disconnected mode. If your organization doesn't have a license, contact HiddenLayer for more information. See Hybrid and Disconnected Modes for information about Model Scanner Disconnected mode.

• AIDR License Key: HiddenLayer Support will provide you with a license key. This key is required to start the LLM proxy container, and it will not run without a valid key. The license can be set as an environment variable and the installer will not run without the license being set as a value.
• Credentials to download AIDR container: Credentials for the HiddenLayer container repository are required to download the appropriate images. These can also be obtained from HiddenLayer Support or from your HiddenLayer technical contact.

Tools

The following tools are required for deploying AIDR for GenAI.

• Docker Desktop: Docker Desktop is used to deploy the container to your Kubernetes cluster.

• kubectl: The official Kubernetes CLI tool, used to issue commands to your Kubernetes cluster.

• Kubernetes cluster

  • For cloud deployments, you need kubectl access to the cluster.
  • For local deployment, use Minikube (or something similar) that can run on your local system.

Resource Guidelines

The following are resource recommendations for the virtual machine running the LLM Proxy in a production environment. Azure AKS is used as an example. These resource recommendations can be applied to any virtual environment. For information about an Azure AKS Dv3 virtual machine, see the Azure documentation.

CPU and Memory

Scaling Recommendations

AIDR is horizontally scalable. The latency and throughput for each replica depends on many factors in the deployed environment, including underlying node type, network conditions, and resource contention.

To make the best use of your underlying hardware, we recommend the following:

1. Replica Count

   • Allocate 8 Kubernetes CPU units for each replica
   • Allocate as many replicas as 8 CPU replicas can fit on to the node

   Example

   If the underlying node type is Azure’s Standard_D32_v3, we recommend setting the following Kubernetes parameters:

   replicas:
     min: 4
     max: 4
   resources:
     requests:
       cpu: 8
       memory: 4096Mi

2. Thread Count per Replica

   • Set the environment variable OMP_NUM_THREADS: 8
   Replica Count

   This value will only improve performance if the guidance in the previous step (Replica Count) is applied.