Databricks Foundation Model Serving

AI Runtime Security can be configured to route traffic to a Databricks Foundation Model Serving instance.

Databricks Foundation Model Serving uses the OpenAI API format regardless of the underlying model being served (for example, GPT, Gemini, or other supported foundation models).

Replace <name> with the Databricks Foundation Model Serving instance name. Example: databricks-prod.
Replace <route_prefix> with the route prefix for the model used. Example: /serving-endpoint/gpt-model.
Replace <base_url> with the URL for the instance. Example: https://upstream.provider.com.
Set the enrich response to true to return the upstream LLM response enriched with HiddenLayer’s detections and analysis. Set to false to disable the HiddenLayer’s detections and analysis.
Replace <api_key> with a Databricks API key.

HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS="<name>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ROUTE_PREFIX="<route_prefix>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_BASE_URL="<base_url>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ENRICH_RESPONSE="<true|false>"
HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_API_KEY="<api_key>"

Environment Key	Required	Description
`HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS`	False	The name of the Databricks Foundation Model Serving instance.
`HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ROUTE_PREFIX`	False	The route prefix for the model used.
`HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_BASE_URL`	False	The base URL for the instance.
`HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_ENRICH_RESPONSE`	False	Set AI Runtime Security as a forward or reverse proxy.
`HL_LLM_PROXY_OPENAI_ROUTES_DATABRICKS_API_KEY`	False	Databricks API key.