Deploy and operate
Documentation

Version Latest

Kubernetes

Run every plane of Neural Inverse Govern on Kubernetes with the keystone Helm chart, on premises or in your own cloud account.

The keystone Helm chart (deploy/helm/keystone in the release) runs every Govern plane as its own workload. It supersedes the older ni-auth chart, which runs the server only.

Requirements

  • Kubernetes 1.29 or later, and Helm 3
  • PostgreSQL 15 or later reachable from the cluster (TLS required by default)
  • The Govern images in a registry your cluster can pull from: keystone-server, keystone-console, keystone-plane and keystone-migrate, tagged with the release commit
  • For more than one server replica, a storage class that supports ReadWriteMany (for example NFS, CephFS, EFS, Azure Files or Filestore), or an S3-compatible bucket

What the chart deploys

PlaneWorkloadBinaryPortProbes
Controlkeystone-serverni-auth8080 (API), 9090 (worker API)readiness /readyz/selfguard, liveness /healthz
Consolekeystone-consoleNext.js3000/signin
Telemetrykeystone-ingestkeystone-ingest8081/readyz, /healthz
Datakeystone-bridgekeystone-bridge8082, 3129/readyz, /healthz
Compliancekeystone-complykeystone-comply9093/readyz, /healthz

Each plane gets its own ServiceAccount (no token mounted), Service, PodDisruptionBudget and CPU autoscaler; topology spread across zones (required) and hosts (preferred); a non-root security context with a read-only root file system and every capability dropped; rolling updates with maxUnavailable: 0; and NetworkPolicies that deny ingress by default.

Before every install and upgrade, hooks:

  1. create or update the three database roles (databaseRoles.bootstrap, run as the database superuser or the Amazon RDS master user);
  2. run the migrations as keystone_owner from the keystone-migrate image of the same release;
  3. run ni-auth cell init, which is idempotent.

The planes then connect as keystone_app (DATABASE_URL, under row-level security) and keystone_platform (DATABASE_PLATFORM_URL). Only the hooks mount the owner and superuser credentials. If your roles are managed outside the chart, set databaseRoles.bootstrap: false.

Create the Secrets

Secrets never go in values. Either create the Kubernetes Secrets named in secrets.* yourself, or turn on externalSecrets.enabled to sync them from AWS Secrets Manager with the External Secrets Operator.

ValueDefault nameHolds
secrets.serverkeystone-server-envEvery server key (KMS_MASTER_KEY, the NI_AUTH_* peppers and encryption keys, NI_OBSERVABILITY_SIGNING_KEY, and the rest listed in Deploy and operate)
secrets.databasekeystone-databaseDATABASE_URL (keystone_app) and DATABASE_PLATFORM_URL (keystone_platform)
secrets.databaseAdminkeystone-database-adminDATABASE_OWNER_URL (keystone_owner); with role bootstrap also KEYSTONE_SUPERUSER_URL and KEYSTONE_OWNER_PASSWORD, KEYSTONE_APP_PASSWORD, KEYSTONE_PLATFORM_PASSWORD. Hooks only.
secrets.consolekeystone-console-envThe console's settings
secrets.ingestkeystone-ingest-envNI_INGEST_SERVICE_TOKEN
secrets.bridgekeystone-bridge-envThe data plane's settings
secrets.complykeystone-comply-envThe compliance plane's settings
secrets.oidcKeykeystone-oidcoidc_private_key (PEM)

Set the values

ValueSet to
image.registry, image.tagYour registry and the release commit. The tag is required and the chart refuses latest.
publicApiUrlThe public API URL, https://api.<your domain>. It must be HTTPS.
cell.*cell.id stays local for a self-hosted installation.
database.host, database.name, database.sslmodeYour PostgreSQL server; sslmode defaults to require.
database.poolPerPodConnections per server pod (default 50). Keep pods times this below PostgreSQL's max_connections.
ingress.enabled, ingress.className, ingress.hosts, ingress.tlsYour Ingress. hosts has console, api, auth and terminal.
networkPolicy.allowedCidrsThe networks your load balancers reach the pods from (default 10.0.0.0/8).

The console host is the address people and clients are given, so the Ingress sends its /.well-known/* and /oauth2/* paths to the server and the rest to the console. A load balancer in front of your Ingress must pass the Host header through unchanged.

Object storage

Exports and artifacts (audit and Observe exports, agent release binaries) live in shared object storage, so any server replica can serve them. Set exactly one; the chart refuses to render without a store.

  • A shared directory (the default, and the choice for air-gapped installations): a ReadWriteMany volume every server replica mounts. Use objectStorage.local.existingClaim, or let the chart create one with objectStorage.local.create (100Gi by default, kept on uninstall). ReadWriteOnce works only with a single server replica.
  • S3 or an S3-compatible store: objectStorage.s3.bucket, kmsKeyId (required), prefix (default keystone/), and endpoint for MinIO, Ceph or another S3-compatible store, with objectStorage.local.create.enabled=false. Never make the bucket public: downloads are streamed through the API, which checks access and audits each one.

Replicas

By default the server runs 3 replicas (autoscaling 3 to 9, at least 2 available), and the console, ingest, bridge and compliance planes 2 each. Adjust server.replicas, server.hpa and server.pdb and the matching values of the other planes.

Render and install

Render the chart and review it before you install:

helm template keystone deploy/helm/keystone --kube-version 1.31.0 \
  --set image.tag=<sha> --set publicApiUrl=https://api.example.com

Then install or upgrade with your values file:

helm upgrade --install keystone deploy/helm/keystone -f values.yaml

After the install, kubectl get pods,statefulsets,pvc,hpa,pdb in the release namespace shows the planes.

Optional components

These are off by default; with them off, the chart renders exactly as without them.

ValueTurns on
telemetryPipeline.enabledThe built-in telemetry pipeline: the bridge becomes a StatefulSet with a write-ahead log on its own volume, and ingest writes hourly PostgreSQL partitions and compacts them to Parquet. Needs exactly one cold store: an S3 bucket or a ReadWriteMany directory.
streams.enabledThe Streams brokers as a StatefulSet, coordinated through the main database.
dns.enabledThe DNS name servers.

Each needs its own Secret first; the chart README in the release lists the keys.

Connection pooling

With many replicas, use PgBouncer in session mode and point both DATABASE_URL and DATABASE_PLATFORM_URL at it. Transaction mode is not supported, because the tenant of each connection is a session setting.

Rollback and uninstall

helm history keystone
helm rollback keystone

Database migrations are not rolled back by Helm. Take a backup before every upgrade and restore it rather than migrating down.

helm uninstall keystone

Volumes the chart created for object storage are kept on uninstall; delete them yourself if you no longer need the data.

Last reviewed 17 October 2026View as Markdown

Kubernetes · Docs · Neural Inverse