Skip to content
AI Detection & Response (...
/LLM Proxy API/
/
Proxy HuggingFace TGI End...

LLM Proxy (1)

HiddenLayer LLM-Proxy is a service that provides detection and response capabilities for Generative AI solutions in the HiddenLayer AI Security Platform.

Locally deployed AIDR proxy in Hybrid mode
  • The LLM Proxy API endpoint is supported only for a locally deployed AIDR proxy running in Hybrid mode. This API is not supported in HiddenLayer SaaS.
  • The documentation is presented here on the documentation portal to help reduce confusion between a locally running proxy and the SaaS proxy.
  • For more API documentation, see the HiddenLayer Developer Portal.
Download OpenAPI description
llm_proxy_api.json
llm_proxy_api.yaml
Languages
Servers
Mock server
https://docs.hiddenlayer.ai/_mock/docs/products/aidr-g/llm_proxy_api/
Self-hosted server
https://<YOUR-SELF-HOSTED-AIDR-INSTANCE-ENDPOINT>/

Health and Metrics

Operations

OpenAI

Operations

Azure

Operations

HuggingFace TGI

Operations

Reverse Proxy HuggingFace TGI endpoint

Request

Generate completions for a HuggingFace TGI endpoint

Path
namestringrequired

The name of the TGI endpoint defined in service settings

Headers
HL-Project-Idstring

The ID or alias for the Project to govern the request processing.

Example: internal-search-chatbot
X-Requester-Idstring

The identifier for the requester to be used if MLDR is enabled

X-LLM-Block-Unsafeboolean

Whether to block unsafe input and output

Default false
X-LLM-Block-Unsafe-Inputboolean

Whether to block unsafe input

Default false
X-LLM-Block-Unsafe-Outputboolean

Whether to block unsafe output

Default false
X-LLM-Skip-Prompt-Injection-Detectionboolean

Whether to skip prompt injection detection

Default false
X-LLM-Block-Prompt-Injectionboolean

Whether to block prompt injection

Default false
X-LLM-Prompt-Injection-Scan-Typestring(HiddenLayerAnalysisPromptInjectionEnum)

The type of prompt injection scan to use

Default quick
Enum"quick""full"
Example: quick
X-LLM-Skip-Input-DOS-Detectionboolean

Whether to skip input denial of service detection

Default false
X-LLM-Block-Input-DOS-Detectionboolean

Whether to block input denial of service detection

Default false
X-LLM-Input-DOS-Detection-Thresholdstring

The threshold for input denial of service detection

Default 4096
X-LLM-Skip-Input-PII-Detectionboolean

Whether to skip input personally identifiable information detection

Default false
X-LLM-Skip-Output-PII-Detectionboolean

Whether to skip output personally identifiable information detection

Default false
X-LLM-Block-Input-PIIboolean

Whether to block input personally identifiable information detection

Default false
X-LLM-Block-Output-PIIboolean

Whether to block output personally identifiable information detection

Default false
X-LLM-Redact-Input-PIIboolean

Whether to redact input personally identifiable information

Default false
X-LLM-Redact-Output-PIIboolean

Whether to redact output personally identifiable information

Default false
X-LLM-Redact-Typestring(HiddenLayerAnalysisRedactTypeEnum)

The type of redaction to use

Default entity
Enum"entity""strict"
Example: entity
X-LLM-Entity-Typestring(HiddenLayerAnalysisEntityTypeEnum)

The type of entity to redact

Default strict
Enum"strict""all"
Example: strict
X-LLM-Skip-Input-Code-Detectionboolean

Whether to skip input code detection

Default false
X-LLM-Skip-Output-Code-Detectionboolean

Whether to skip output code detection

Default false
X-LLM-Block-Input-Code-Detectionboolean

Whether to block input code detection

Default false
X-LLM-Block-Output-Code-Detectionboolean

Whether to block output code detection

Default false
X-LLM-Skip-Guardrail-Detectionboolean

Whether to skip guardrail detection

Default false
X-LLM-Block-Guardrail-Detectionboolean

Whether to block guardrail detection

Default false
X-LLM-Skip-Input-URL-Detectionboolean

Whether to skip input URL detection

Default false
X-LLM-Skip-Output-URL-Detectionboolean

Whether to skip output URL detection

Default false
Bodyapplication/jsonrequired
modelstring

The model to use for completions

messagesArray of objects
temperaturenumber

The temperature to use for completions

max_tokensnumber

The maximum number of tokens to generate

top_pnumber

The top p value to use for completions

frequency_penaltynumber

The frequency penalty to use for completions

presence_penaltynumber

The presence penalty to use for completions

curl -i -X POST \
  'https://docs.hiddenlayer.ai/_mock/docs/products/aidr-g/llm_proxy_api/tgi{name}/v1/chat/completions' \
  -H 'Content-Type: application/json' \
  -H 'HL-Project-Id: internal-search-chatbot' \
  -H 'X-LLM-Block-Guardrail-Detection: false' \
  -H 'X-LLM-Block-Input-Code-Detection: false' \
  -H 'X-LLM-Block-Input-DOS-Detection: false' \
  -H 'X-LLM-Block-Input-PII: false' \
  -H 'X-LLM-Block-Output-Code-Detection: false' \
  -H 'X-LLM-Block-Output-PII: false' \
  -H 'X-LLM-Block-Prompt-Injection: false' \
  -H 'X-LLM-Block-Unsafe: false' \
  -H 'X-LLM-Block-Unsafe-Input: false' \
  -H 'X-LLM-Block-Unsafe-Output: false' \
  -H 'X-LLM-Entity-Type: strict' \
  -H 'X-LLM-Input-DOS-Detection-Threshold: 4096' \
  -H 'X-LLM-Prompt-Injection-Scan-Type: quick' \
  -H 'X-LLM-Redact-Input-PII: false' \
  -H 'X-LLM-Redact-Output-PII: false' \
  -H 'X-LLM-Redact-Type: entity' \
  -H 'X-LLM-Skip-Guardrail-Detection: false' \
  -H 'X-LLM-Skip-Input-Code-Detection: false' \
  -H 'X-LLM-Skip-Input-DOS-Detection: false' \
  -H 'X-LLM-Skip-Input-PII-Detection: false' \
  -H 'X-LLM-Skip-Input-URL-Detection: false' \
  -H 'X-LLM-Skip-Output-Code-Detection: false' \
  -H 'X-LLM-Skip-Output-PII-Detection: false' \
  -H 'X-LLM-Skip-Output-URL-Detection: false' \
  -H 'X-LLM-Skip-Prompt-Injection-Detection: false' \
  -H 'X-Requester-Id: string' \
  -d '{
    "model": "tgi-model",
    "messages": [
      {
        "role": "user",
        "content": "Hello World!"
      }
    ],
    "temperature": 1,
    "max_tokens": 256,
    "top_p": 1,
    "frequency_penalty": 0,
    "presence_penalty": 0
  }'

Responses

Successful HuggingFace TGI response

Bodyapplication/json
choicesArray of objects
creatednumber

The timestamp of the completion

idnull or string

The id of the completion

modelstring

The model used for the completion

objectstring

The object of the completion

usageobject
Response
application/json
{
  "choices": [
    { … }
  ],
  "created": 1640000000,
  "id": null,
  "model": "tgi-model",
  "object": "chat.completion",
  "usage": {
    "completion_tokens": 17,
    "prompt_tokens": 15,
    "total_tokens": 32
  }
}

Proxy HuggingFace TGI Endpoint

Request

Proxy chat completions for a given HuggingFace TGI endpoint and return response and analysis

Path
namestringrequired

The name of the TGI endpoint defined in service settings

Headers
HL-Project-Idstring

The ID or alias for the Project to govern the request processing.

Example: internal-search-chatbot
X-Requester-Idstring

The identifier for the requester to be used if MLDR is enabled

X-LLM-Block-Unsafeboolean

Whether to block unsafe input and output

Default false
X-LLM-Block-Unsafe-Inputboolean

Whether to block unsafe input

Default false
X-LLM-Block-Unsafe-Outputboolean

Whether to block unsafe output

Default false
X-LLM-Skip-Prompt-Injection-Detectionboolean

Whether to skip prompt injection detection

Default false
X-LLM-Block-Prompt-Injectionboolean

Whether to block prompt injection

Default false
X-LLM-Prompt-Injection-Scan-Typestring(HiddenLayerAnalysisPromptInjectionEnum)

The type of prompt injection scan to use

Default quick
Enum"quick""full"
Example: quick
X-LLM-Skip-Input-DOS-Detectionboolean

Whether to skip input denial of service detection

Default false
X-LLM-Block-Input-DOS-Detectionboolean

Whether to block input denial of service detection

Default false
X-LLM-Input-DOS-Detection-Thresholdstring

The threshold for input denial of service detection

Default 4096
X-LLM-Skip-Input-PII-Detectionboolean

Whether to skip input personally identifiable information detection

Default false
X-LLM-Skip-Output-PII-Detectionboolean

Whether to skip output personally identifiable information detection

Default false
X-LLM-Block-Input-PIIboolean

Whether to block input personally identifiable information detection

Default false
X-LLM-Block-Output-PIIboolean

Whether to block output personally identifiable information detection

Default false
X-LLM-Redact-Input-PIIboolean

Whether to redact input personally identifiable information

Default false
X-LLM-Redact-Output-PIIboolean

Whether to redact output personally identifiable information

Default false
X-LLM-Redact-Typestring(HiddenLayerAnalysisRedactTypeEnum)

The type of redaction to use

Default entity
Enum"entity""strict"
Example: entity
X-LLM-Entity-Typestring(HiddenLayerAnalysisEntityTypeEnum)

The type of entity to redact

Default strict
Enum"strict""all"
Example: strict
X-LLM-Skip-Input-Code-Detectionboolean

Whether to skip input code detection

Default false
X-LLM-Skip-Output-Code-Detectionboolean

Whether to skip output code detection

Default false
X-LLM-Block-Input-Code-Detectionboolean

Whether to block input code detection

Default false
X-LLM-Block-Output-Code-Detectionboolean

Whether to block output code detection

Default false
X-LLM-Skip-Guardrail-Detectionboolean

Whether to skip guardrail detection

Default false
X-LLM-Block-Guardrail-Detectionboolean

Whether to block guardrail detection

Default false
X-LLM-Skip-Input-URL-Detectionboolean

Whether to skip input URL detection

Default false
X-LLM-Skip-Output-URL-Detectionboolean

Whether to skip output URL detection

Default false
Bodyapplication/jsonrequired
modelstring

The model to use for completions

messagesArray of objects
temperaturenumber

The temperature to use for completions

max_tokensnumber

The maximum number of tokens to generate

top_pnumber

The top p value to use for completions

frequency_penaltynumber

The frequency penalty to use for completions

presence_penaltynumber

The presence penalty to use for completions

curl -i -X POST \
  'https://docs.hiddenlayer.ai/_mock/docs/products/aidr-g/llm_proxy_api/api/v1/proxy/tgi/{name}' \
  -H 'Content-Type: application/json' \
  -H 'HL-Project-Id: internal-search-chatbot' \
  -H 'X-LLM-Block-Guardrail-Detection: false' \
  -H 'X-LLM-Block-Input-Code-Detection: false' \
  -H 'X-LLM-Block-Input-DOS-Detection: false' \
  -H 'X-LLM-Block-Input-PII: false' \
  -H 'X-LLM-Block-Output-Code-Detection: false' \
  -H 'X-LLM-Block-Output-PII: false' \
  -H 'X-LLM-Block-Prompt-Injection: false' \
  -H 'X-LLM-Block-Unsafe: false' \
  -H 'X-LLM-Block-Unsafe-Input: false' \
  -H 'X-LLM-Block-Unsafe-Output: false' \
  -H 'X-LLM-Entity-Type: strict' \
  -H 'X-LLM-Input-DOS-Detection-Threshold: 4096' \
  -H 'X-LLM-Prompt-Injection-Scan-Type: quick' \
  -H 'X-LLM-Redact-Input-PII: false' \
  -H 'X-LLM-Redact-Output-PII: false' \
  -H 'X-LLM-Redact-Type: entity' \
  -H 'X-LLM-Skip-Guardrail-Detection: false' \
  -H 'X-LLM-Skip-Input-Code-Detection: false' \
  -H 'X-LLM-Skip-Input-DOS-Detection: false' \
  -H 'X-LLM-Skip-Input-PII-Detection: false' \
  -H 'X-LLM-Skip-Input-URL-Detection: false' \
  -H 'X-LLM-Skip-Output-Code-Detection: false' \
  -H 'X-LLM-Skip-Output-PII-Detection: false' \
  -H 'X-LLM-Skip-Output-URL-Detection: false' \
  -H 'X-LLM-Skip-Prompt-Injection-Detection: false' \
  -H 'X-Requester-Id: string' \
  -d '{
    "model": "tgi-model",
    "messages": [
      {
        "role": "user",
        "content": "Hello World!"
      }
    ],
    "temperature": 1,
    "max_tokens": 256,
    "top_p": 1,
    "frequency_penalty": 0,
    "presence_penalty": 0
  }'

Responses

Successful chat completions and analysis response

Bodyapplication/json
HiddenLayerProxyHuggingFaceTGIResponse (object)
Response
application/json
{
  "response": {
    "choices": [ … ],
    "created": 0,
    "id": null,
    "model": "string",
    "object": "string",
    "usage": { … }
  },
  "provider": "huggingface-tgi",
  "model": "tgi-model",
  "verdict": true,
  "categories": {
    "unsafe_input": true,
    "unsafe_output": true,
    "prompt_injection": true,
    "input_dos": true,
    "input_pii": true,
    "output_pii": true,
    "input_code": true,
    "output_code": true,
    "guardrail": true
  },
  "results": {
    "input_block_list_results": { … },
    "prompt_injection_classifier_results": [ … ],
    "input_dos_results": { … },
    "input_pii_results": { … },
    "output_pii_results": { … },
    "input_code_results": { … },
    "output_code_results": { … },
    "guardrail_results": { … },
    "input_urls": { … },
    "output_urls": { … }
  },
  "policy": {
    "block_unsafe": false,
    "block_unsafe_input": false,
    "block_unsafe_output": true,
    "skip_prompt_injection_detection": true,
    "block_prompt_injection": true,
    "prompt_injection_scan_type": "quick",
    "skip_input_pii_detection": true,
    "skip_output_pii_detection": true,
    "block_input_pii": true,
    "block_output_pii": true,
    "redact_input_pii": true,
    "redact_output_pii": true,
    "redact_type": "entity",
    "entity_type": "strict",
    "skip_input_code_detection": true,
    "skip_output_code_detection": true,
    "block_input_code_detection": true,
    "block_output_code_detection": true,
    "skip_guardrail_detection": true,
    "block_guardrail_detection": true,
    "skip_input_url_detection": true,
    "skip_output_url_detection": true,
    "skip_input_dos_detection": true,
    "block_input_dos_detection": true,
    "input_dos_detection_threshold": 4096
  },
  "frameworks": {
    "mitre": [ … ],
    "owasp": [ … ]
  },
  "elapsed_ms": 0,
  "upstream_elapsed_ms": 0
}