Skip to content
AI Runtime Security/
/
Databricks Foundation Mod...

Databricks Foundation Model Serving

AI Runtime Security can be configured to route traffic to a Databricks Foundation Model Serving instance.

Databricks Foundation Model Serving uses the OpenAI API format regardless of the underlying model being served (for example, GPT, Gemini, or other supported foundation models).

  • Replace <name> with the Databricks Foundation Model Serving instance name. Example: databricks-prod.
  • Replace <route_prefix> with the route prefix for the model used. Example: /serving-endpoint/gpt-model.
  • Replace <base_url> with the URL for the instance. Example: https://upstream.provider.com.
  • Set the enrich response to true to return the upstream LLM response enriched with HiddenLayer’s detections and analysis. Set to false to disable the HiddenLayer’s detections and analysis.
  • Replace <api_key> with a Databricks API key.
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS="<name>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ROUTE_PREFIX="<route_prefix>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_BASE_URL="<base_url>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ENRICH_RESPONSE="<true|false>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_API_KEY="<api_key>"
Environment KeyDefaultRequiredDescription
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKSFalseThe name of the Databricks Foundation Model Serving instance.
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ROUTE_PREFIXFalseThe route prefix for the model used.
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_BASE_URLFalseThe base URL for the instance.
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ENRICH_RESPONSEFalseSet AI Runtime Security as a forward or reverse proxy.
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_API_KEYFalseDatabricks API key.

Azure Key Vault

LLM Sandbox Test