---
title: "Kubernetes"
---

# Kubernetes

Run every plane of Neural Inverse Govern on Kubernetes with the keystone Helm chart, on premises or in your own cloud account.

The `keystone` Helm chart (`deploy/helm/keystone` in the release) runs every Govern plane as its own workload. It supersedes the older `ni-auth` chart, which runs the server only.

## Requirements

- Kubernetes 1.29 or later, and Helm 3
- PostgreSQL 15 or later reachable from the cluster (TLS required by default)
- The Govern images in a registry your cluster can pull from: `keystone-server`, `keystone-console`, `keystone-plane` and `keystone-migrate`, tagged with the release commit
- For more than one server replica, a storage class that supports ReadWriteMany (for example NFS, CephFS, EFS, Azure Files or Filestore), or an S3-compatible bucket

## What the chart deploys

| Plane | Workload | Binary | Port | Probes |
|---|---|---|---|---|
| Control | `keystone-server` | `ni-auth` | 8080 (API), 9090 (worker API) | readiness `/readyz/selfguard`, liveness `/healthz` |
| Console | `keystone-console` | Next.js | 3000 | `/signin` |
| Telemetry | `keystone-ingest` | `keystone-ingest` | 8081 | `/readyz`, `/healthz` |
| Data | `keystone-bridge` | `keystone-bridge` | 8082, 3129 | `/readyz`, `/healthz` |
| Compliance | `keystone-comply` | `keystone-comply` | 9093 | `/readyz`, `/healthz` |

Each plane gets its own ServiceAccount (no token mounted), Service, PodDisruptionBudget and CPU autoscaler; topology spread across zones (required) and hosts (preferred); a non-root security context with a read-only root file system and every capability dropped; rolling updates with `maxUnavailable: 0`; and NetworkPolicies that deny ingress by default.

Before every install and upgrade, hooks:

1. create or update the three database roles (`databaseRoles.bootstrap`, run as the database superuser or the Amazon RDS master user);
2. run the migrations as `keystone_owner` from the `keystone-migrate` image of the same release;
3. run `ni-auth cell init`, which is idempotent.

The planes then connect as `keystone_app` (`DATABASE_URL`, under row-level security) and `keystone_platform` (`DATABASE_PLATFORM_URL`). Only the hooks mount the owner and superuser credentials. If your roles are managed outside the chart, set `databaseRoles.bootstrap: false`.

## Create the Secrets

Secrets never go in values. Either create the Kubernetes Secrets named in `secrets.*` yourself, or turn on `externalSecrets.enabled` to sync them from AWS Secrets Manager with the External Secrets Operator.

| Value | Default name | Holds |
|---|---|---|
| `secrets.server` | `keystone-server-env` | Every server key (`KMS_MASTER_KEY`, the `NI_AUTH_*` peppers and encryption keys, `NI_OBSERVABILITY_SIGNING_KEY`, and the rest listed in [Deploy and operate](https://neuralinverse.com/docs/deploy/)) |
| `secrets.database` | `keystone-database` | `DATABASE_URL` (`keystone_app`) and `DATABASE_PLATFORM_URL` (`keystone_platform`) |
| `secrets.databaseAdmin` | `keystone-database-admin` | `DATABASE_OWNER_URL` (`keystone_owner`); with role bootstrap also `KEYSTONE_SUPERUSER_URL` and `KEYSTONE_OWNER_PASSWORD`, `KEYSTONE_APP_PASSWORD`, `KEYSTONE_PLATFORM_PASSWORD`. Hooks only. |
| `secrets.console` | `keystone-console-env` | The console's settings |
| `secrets.ingest` | `keystone-ingest-env` | `NI_INGEST_SERVICE_TOKEN` |
| `secrets.bridge` | `keystone-bridge-env` | The data plane's settings |
| `secrets.comply` | `keystone-comply-env` | The compliance plane's settings |
| `secrets.oidcKey` | `keystone-oidc` | `oidc_private_key` (PEM) |

## Set the values

| Value | Set to |
|---|---|
| `image.registry`, `image.tag` | Your registry and the release commit. The tag is required and the chart refuses `latest`. |
| `publicApiUrl` | The public API URL, `https://api.<your domain>`. It must be HTTPS. |
| `cell.*` | `cell.id` stays `local` for a self-hosted installation. |
| `database.host`, `database.name`, `database.sslmode` | Your PostgreSQL server; `sslmode` defaults to `require`. |
| `database.poolPerPod` | Connections per server pod (default 50). Keep pods times this below PostgreSQL's `max_connections`. |
| `ingress.enabled`, `ingress.className`, `ingress.hosts`, `ingress.tls` | Your Ingress. `hosts` has `console`, `api`, `auth` and `terminal`. |
| `networkPolicy.allowedCidrs` | The networks your load balancers reach the pods from (default `10.0.0.0/8`). |

The console host is the address people and clients are given, so the Ingress sends its `/.well-known/*` and `/oauth2/*` paths to the server and the rest to the console. A load balancer in front of your Ingress must pass the `Host` header through unchanged.

### Object storage

Exports and artifacts (audit and Observe exports, agent release binaries) live in shared object storage, so any server replica can serve them. Set exactly one; the chart refuses to render without a store.

- **A shared directory** (the default, and the choice for air-gapped installations): a ReadWriteMany volume every server replica mounts. Use `objectStorage.local.existingClaim`, or let the chart create one with `objectStorage.local.create` (100Gi by default, kept on uninstall). ReadWriteOnce works only with a single server replica.
- **S3 or an S3-compatible store**: `objectStorage.s3.bucket`, `kmsKeyId` (required), `prefix` (default `keystone/`), and `endpoint` for MinIO, Ceph or another S3-compatible store, with `objectStorage.local.create.enabled=false`. Never make the bucket public: downloads are streamed through the API, which checks access and audits each one.

### Replicas

By default the server runs 3 replicas (autoscaling 3 to 9, at least 2 available), and the console, ingest, bridge and compliance planes 2 each. Adjust `server.replicas`, `server.hpa` and `server.pdb` and the matching values of the other planes.

## Render and install

Render the chart and review it before you install:

```bash
helm template keystone deploy/helm/keystone --kube-version 1.31.0 \
  --set image.tag=<sha> --set publicApiUrl=https://api.example.com
```

Then install or upgrade with your values file:

```bash
helm upgrade --install keystone deploy/helm/keystone -f values.yaml
```

After the install, `kubectl get pods,statefulsets,pvc,hpa,pdb` in the release namespace shows the planes.

## Optional components

These are off by default; with them off, the chart renders exactly as without them.

| Value | Turns on |
|---|---|
| `telemetryPipeline.enabled` | The built-in telemetry pipeline: the bridge becomes a StatefulSet with a write-ahead log on its own volume, and ingest writes hourly PostgreSQL partitions and compacts them to Parquet. Needs exactly one cold store: an S3 bucket or a ReadWriteMany directory. |
| `streams.enabled` | The Streams brokers as a StatefulSet, coordinated through the main database. |
| `dns.enabled` | The DNS name servers. |

Each needs its own Secret first; the chart README in the release lists the keys.

## Connection pooling

With many replicas, use PgBouncer in session mode and point both `DATABASE_URL` and `DATABASE_PLATFORM_URL` at it. Transaction mode is not supported, because the tenant of each connection is a session setting.

## Rollback and uninstall

```bash
helm history keystone
helm rollback keystone
```

Database migrations are not rolled back by Helm. Take a backup before every upgrade and restore it rather than migrating down.

```bash
helm uninstall keystone
```

Volumes the chart created for object storage are kept on uninstall; delete them yourself if you no longer need the data.