Skip to main content

​
Example of Setting Up Required Infrastructure via AWS

The platform depends on five external data services that will need to be provisioned manually within your own infrastructure. HiddenLayer has tested and validated these services in AWS, but AWS is not required. You may host the services on any provider or on-premises infrastructure that meets the requirements below. The setup steps in this section are provided as a courtesy for AWS deployments. HiddenLayer is not responsible for provisioning, configuring, or supporting the underlying services, regardless of where you choose to host them. For the AWS-based setup described below, HiddenLayer has validated the following services:
Two separate OpenSearch domains are required: one for the platform services and one for FusionAuth. Do not point both at a single domain (see the OpenSearch section for details).
This section describes the supported configuration for each component. AWS exposes many alternatives (IAM database authentication, fine-grained access control, SASL/SCRAM, RBAC, cluster mode) that are not supported. Deviating from the settings below is not supported and may prevent the platform from operating correctly.

​
Before you start

These apply to all components.
  • Subnet placement is yours to decide. Put these components wherever your own network and security standards dictate. HiddenLayer recommends private subnets with no public endpoint, but nothing in the platform requires it. It only needs to reach the endpoints you hand it.
  • Availability Zone and subnet counts are a service constraint, not a preference. Use three subnets in three different Availability Zones. MSK accepts only two or three, and the platform’s replication settings require three brokers (see the MSK section), so three subnets is the configuration that satisfies every component.
  • Know every CIDR block your VPC carries. A VPC can have more than one, and worker nodes are frequently spread across subnets in several of them. Security group rules below are written in terms of “your VPC CIDR”. If your VPC has multiple CIDRs, add a rule for each one. A client in an unlisted range does not receive a permission error; it simply hangs until it times out, which typically presents as an intermittent failure that depends on where a pod happened to be scheduled.
  • Use security groups to control access. For most components, network reachability is the primary control over who can connect. Scope each security group’s rules to only the address ranges your platform runs in, and consider this carefully before placing any component on a public network.
  • Encryption at rest is on everywhere. Each component below enables it. The AWS-owned/managed key is sufficient; use a customer-managed KMS key if your policies require one.
  • Sizing. The instance sizes quoted in each section are the smallest the service permits and are intended for a minimal deployment, not a production workload. Size for your own throughput and retention before going live; where a setting is painful to change later, it is called out.

​
Amazon RDS for PostgreSQL

Console: RDS → Databases → Create database. Reference: Creating an Amazon RDS DB instance.

​
Create the instance

​
Authentication

Use password authentication with a master user, and set the password yourself at creation time. You will supply this static username and password to the installer. Do not select “Manage master credentials in AWS Secrets Manager”. The platform expects a static password value and cannot pick up rotated credentials from Secrets Manager, which will cause all services to lose their database connection when the credential rotates. Do not enable IAM database authentication. The platform does not support short-lived IAM credentials for PostgreSQL connections. The RDS master user is granted rds_superuser, which provides the privileges the platform requires.

​
Parameter group

Create a custom DB parameter group (family postgres17) and attach it at creation. Defaults are not sufficient. Both are static parameters. On a new instance they are applied at first boot; changing them on a running instance requires a reboot before they take effect.
Do not set rds.force_ssl. Some platform components do not negotiate TLS to PostgreSQL, and enabling this parameter will prevent them from connecting. Connections are unencrypted in transit unless a given client opts in, so keep the security group tightly scoped.
Reference: Working with parameter groups, Using pg_cron.

​
Security group

Inbound: TCP 5432 from your VPC CIDR (one rule per CIDR block). Nothing else needs to be done inside the database. The platform creates its own per-service databases and enables the pg_cron extension during installation. You only need to provide the empty instance, the master credentials, and the parameter group above. Record: writer endpoint, port, master username, master password, initial database name.

​
Amazon OpenSearch Service

Two separate domains are required: one for the platform services, one for FusionAuth. Do not point both at a single domain. The two workloads are a poor fit for one cluster, and the failure modes run in the direction you least want:
  • Availability tiers differ. FusionAuth’s search backs login. If the cluster goes red, a circuit breaker trips, or a node runs out of heap, users cannot authenticate. Platform ingest is bursty and unbounded, and the reporting assistant generates queries dynamically, so a single expensive aggregation can exhaust a data node. On one cluster, a reporting query becomes an authentication outage.
  • Resource contention is unavoidable. Field data, query caches, and thread pools are shared per node. Nothing on a small cluster isolates a low-volume latency-sensitive index from continuous bulk writes and daily index rollover.
  • Maintenance couples the two planes. Any change to the platform domain (resizing an instance type, changing storage, upgrading the engine) runs as a blue/green migration lasting tens of minutes. Colocated, every one of those becomes a risk window for logins, and both planes are forced onto a single maintenance schedule.
  • Snapshot and restore granularity. Recovering platform data after a bad migration would otherwise mean restoring a cluster that also holds your user records.
Separating them costs one small domain. Colocating them makes authentication share a fate with reporting. Console: OpenSearch Service → Domains → Create domain. Reference: Creating and managing domains.

​
Domain 1: Platform services

Size this domain for your event volume and retention. The platform creates rollover index families with daily rollover and 30-day retention, so index and shard counts grow with traffic. Expect this to be the larger of the two domains by a wide margin.

​
Domain 2: FusionAuth

FusionAuth’s index footprint is small (on the order of 1–2 GB), so this domain is sized for availability rather than capacity.
The engine version differs from Domain 1. FusionAuth supports a narrower range of search engine versions than the platform services do, so this domain runs OpenSearch 2.x while the platform domain runs 3.x. Running them as separate domains is what makes that possible.
Three nodes across three Availability Zones is the point of this domain: it keeps login working through the loss of a single node or AZ, which a single-node domain cannot. The instance size is deliberately small; this workload is a rounding error next to the platform’s.

​
Shared settings for both domains

The guidance below applies to each domain independently.
Subnets and public access
If you select 1 AZ, attach the domain to exactly one subnet. Multi-AZ requires one subnet per AZ and a node count that is a multiple of the AZ count. These are service constraints rather than recommendations. If you choose a public endpoint, understand that it combines with the authentication setting below: the domain performs no authentication, so a public endpoint makes it reachable and fully usable from the internet by anyone who learns the address. A VPC endpoint is strongly recommended for that reason, but the choice is yours. Reference: VPC support for OpenSearch Service domains.
Authentication
Do not enable fine-grained access control. The platform requires FGAC to be disabled. With FGAC disabled, the domain performs no authentication and no authorization, so any client that can reach the endpoint has full read, write, and administrative access to every index.
Set the domain access policy to allow access without request signing. Because the domain is VPC-only, reachability is governed entirely by its security group; treat that as the security boundary. This is a further reason to keep the two domains apart: with no authentication on either, anything that can reach the FusionAuth domain can read every user record. A separate domain with its own security group is what limits that reach. Two consequences worth planning around:
  • Fine-grained access control cannot be turned off once a domain has been created with it. If you enable it, you will need to replace the domain to reach a supported configuration.
  • Audit logs require fine-grained access control, so they are unavailable in this configuration. Error and slow logs are not affected.
Reference: Fine-grained access control, Identity and access management.
Logging (optional)
Error logs and search/index slow logs can be published to CloudWatch Logs. Enabling this from the console creates the log groups and the required CloudWatch Logs resource policy for you. Slow logs also need per-index thresholds set at runtime before they emit anything. Reference: Monitoring logs.
Security group
Give each domain its own security group, each allowing inbound TCP 443 from your VPC CIDR (one rule per CIDR block). Separate groups let you narrow access to the FusionAuth domain later without touching the platform domain. Record: both domain endpoints (https://…), kept distinct. The platform services endpoint and the FusionAuth endpoint are configured separately.

​
Amazon MSK (Kafka)

Console: MSK → Clusters → Create cluster. Reference: Creating an Amazon MSK cluster.

​
Create the cluster

Three brokers is a requirement, not a recommendation. The platform provisions its Kafka topics with a replication factor of 3, and Kafka cannot create a topic whose replication factor exceeds the number of brokers. Because MSK also requires the broker count to be an exact multiple of the subnet count, the working configuration is three subnets in three Availability Zones with three brokers. Larger clusters must stay on that multiple (6, 9, and so on).
Set broker storage realistically at creation. Storage can be increased later but never decreased, and scaling actions are limited to one every six hours. The minimum the service accepts is 1 GiB, which will fill almost immediately under real traffic.

​
Authentication and encryption

Configure the cluster for unauthenticated access. This is the only supported client configuration. Clients connect on port 9092 (plaintext) or 9094 (TLS). Both listeners are available with this setting. Public access must stay off. MSK only permits public access on clusters that use TLS-only encryption with a SASL authentication method enabled and unauthenticated access disabled, which is incompatible with the configuration above. The cluster must be reachable over your private network. Plan for the operational impact of unauthenticated access: every client connects as the single Kafka principal ANONYMOUS. Per-client authorization is not possible because clients are indistinguishable, and broker logs attribute all activity to that one principal. Combined with the permissive ACL fallback below, any client that can reach a broker can produce to, consume from, create, and delete any topic. Reference: Client authentication, Encryption.

​
Cluster configuration

Create an MSK configuration and select it during cluster creation. The defaults are not suitable.
  • auto.create.topics.enable=false: the installer pre-creates every topic it needs with the correct partition and replication settings. Leaving auto-creation on means a client typo silently creates a misconfigured topic instead of failing.
  • allow.everyone.if.no.acl.found=true: required for unauthenticated clients to be authorized. With no ACLs defined, this grants ANONYMOUS full access.
  • unclean.leader.election.enable=false: prefers durability over availability.
  • default.replication.factor=3 matches the replication factor the platform uses when it creates its own topics. A cluster with fewer than three brokers cannot satisfy it, and topic creation fails during installation. This is why three brokers is a requirement rather than a sizing suggestion.
  • min.insync.replicas=2 lets acks=all producers keep writing through the loss of one broker, while refusing writes once a second is gone.
Reference: Amazon MSK configuration.

​
Security group

Inbound from your VPC CIDR (one rule per CIDR block):

​
After creation

Retrieve the bootstrap broker list from the console (Cluster → View client information) or with:
Use BootstrapBrokerString for plaintext (9092) or BootstrapBrokerStringTls for TLS (9094). Reference: Getting the bootstrap brokers. Record: the comma-separated bootstrap broker list.

​
Amazon ElastiCache (Valkey)

Console: ElastiCache → Valkey caches → Create Valkey cache. Reference: Creating a cache.

​
Create the cache

Cluster mode must stay disabled. The platform uses a standard single-endpoint Redis client. Cluster mode exposes a configuration endpoint, returns slot redirects, and rejects cross-slot multi-key commands, all of which require a cluster-aware client. It is a breaking change, not a configuration toggle.
Valkey is wire-compatible with Redis, so the platform’s Redis clients work unchanged. Redis OSS is also supported if you prefer it.

​
Authentication

Create the cache without an authentication token and without RBAC user groups. This is the only supported configuration, so the cache accepts any client that can reach it on its port. Note that with no token in place, encryption in transit operates in preferred mode, meaning clients may connect with or without TLS. As with the other components, the security group is the real boundary. ElastiCache does not support mutual TLS. If you are migrating from a self-managed Redis that used client certificates, that configuration has no equivalent here. Remove the certificate and key settings rather than trying to port them. Reference: Authenticating with the Valkey AUTH command, In-transit encryption.

​
Parameter group

Create a custom cache parameter group and set: Reference: Configuring engine parameters with parameter groups.

​
Security group

Inbound: TCP 6379 from your VPC CIDR (one rule per CIDR block). Record: primary endpoint address and port.

​
Values to supply during installation

Collect the following and set them in your installer configuration before running the install.

​
Verifying connectivity

​
For Embedded Cluster deployment

After the initial setup is complete, you will be re-directed to the Admin Console.
  • The Self-Hosted AI Security Platform will take time to complete installation after the package is deployed.
  • The Platform Console will be accessible when the status changes to “Ready” as seen in the Admin Console.
Configure HiddenLayer Platform

​
For Helm deployment

Before running the installer, confirm each endpoint is reachable from inside the cluster, not from a workstation. Run a throwaway pod in the target namespace and connect to each service in turn. A timeout rather than an authentication or protocol error almost always means a security group is missing a rule for the CIDR that pod’s node sits in, which is the most common failure in this setup.