---
title: "Troubleshooting"
---

# Troubleshooting

Symptoms, likely causes and fixes for a Govern installation that will not start, cannot reach its database, sheds load, fails to send mail, or reports a broken chain.

Start with three questions: does the process run, can it reach PostgreSQL, and what do the logs and selfguard say?

```bash
# Docker Compose
docker compose ps
docker compose logs server --since 10m | grep -iE "error|panic|fatal|refused"
curl -s http://localhost:8080/healthz

# Kubernetes
kubectl get pods
kubectl logs <server pod> --since=10m | grep -iE "error|panic|fatal|refused"

# Selfguard, even while the database is down
curl -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" https://<your Govern host>/metrics/selfguard
```

## The server does not start

| Symptom | Cause | Fix |
|---|---|---|
| Compose refuses to start and names a variable | A required variable is empty | Set it; generation commands are in [Docker Compose](https://neuralinverse.com/docs/deploy/docker-compose/) |
| The server exits naming a key | A key is missing, too short, all zero bytes or a placeholder (`change-me`, `replace-with`, ...) | Generate a real value. `NI_AUTH_DEPLOY_ENV=development` accepts placeholders for local development only |
| `DATABASE_OWNER_URL is required` from `migrate` | Migrations run only as `keystone_owner` | Set `DATABASE_OWNER_URL`; see [Upgrades and migrations](https://neuralinverse.com/docs/deploy/upgrades/) |
| The server refuses the runtime role | `DATABASE_URL` is a superuser, has `BYPASSRLS`, owns objects, or equals `DATABASE_PLATFORM_URL` | Point `DATABASE_URL` at `keystone_app` and `DATABASE_PLATFORM_URL` at `keystone_platform`; run the role bootstrap |
| The server refuses to start because tables are not isolated | `keystone_tenancy_problems()` returns rows: a migration did not finish, a grant was made by hand, or a partition lacks its parent's privileges | Run the role bootstrap and the migrations as the owner; as the superuser, read `SELECT * FROM keystone_tenancy_problems();`; revoke hand-made grants. Never grant a runtime role more to make it start |
| Refused for a repository password | `DATABASE_URL` or `DATABASE_PLATFORM_URL` uses a development password shipped in the repository | Set real role passwords and run the bootstrap |
| Stops when a KMS-sealed key cannot be opened | `KMS_MASTER_KEY` differs from the one the keys were sealed under | Restore the original `KMS_MASTER_KEY` |
| Migrations stop: "Keystone requires PostgreSQL 15 or later" | PostgreSQL 14 or older | Upgrade PostgreSQL first; nothing was changed |
| "database is dirty at version N" | A migration was interrupted | See the dirty migration steps in [Upgrades and migrations](https://neuralinverse.com/docs/deploy/upgrades/) |
| Stops: audit tiering bucket missing | The ledger already has entries in the cold tier | Configure `NI_AUTH_AUDIT_TIERING_BUCKET` again |
| Stops: backup cannot start | `NI_AUTH_BACKUP_ENCRYPTION_KEY` set, but the key, region or credentials are invalid | Fix them, or unset the key |
| Stops on a client IP setting | `NI_AUTH_CLIENT_IP_HEADER` set without `NI_AUTH_TRUSTED_CDN_PROXIES`, or a catch-all range | See [Client IP addresses](https://neuralinverse.com/docs/deploy/client-ip/) |

## It runs, but requests fail

| Symptom | Cause | Fix |
|---|---|---|
| Sign-in returns `500` | No OIDC private key | Set `NI_AUTH_OIDC_PRIVATE_KEY_FILE` or `NI_AUTH_OIDC_PRIVATE_KEY_PEM` |
| Tokens fail validation | `NI_AUTH_JWT_ISSUER` does not equal `NI_AUTH_PUBLIC_URL` exactly | Make them identical |
| Logs show `no tenant for this statement` (SQLSTATE `42501`) | A code path ran without a tenant and failed closed | Report it with the request id; do not grant the runtime role more |
| `503` with `X-Keystone-Shed: node_overloaded`, or `429` with `tenant_share_exceeded` | Admission control is shedding load | Scale out; check that the shed traffic is batch (`keystone_selfguard_shed_requests_total{priority}`); follow the findings in [Monitoring](https://neuralinverse.com/docs/deploy/monitoring/) |
| `remaining connection slots are reserved` or `too many clients already` | PostgreSQL is out of connections | Find who holds them: `SELECT usename, application_name, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC;`. Lower `NI_AUTH_DB_MAX_OPEN` across replicas, or put PgBouncer (session mode) in front |
| Rate limits and lockouts hit every user at once | All requests appear to come from the proxy | Set `NI_AUTH_TRUSTED_PROXIES` (and CDN settings) |
| Every request in the audit log shows the console host | The console's signed client context is not verified | Set `NI_AUTH_CONSOLE_CONTEXT_KEYS` |

## Database incidents

- **PostgreSQL unreachable.** Selfguard raises `probe.failing/database` and recycles connections; `/readyz` turns not ready. Its history is buffered in memory and written in order when the database returns. Check that PostgreSQL runs and accepts the runtime role's credentials.
- **Sessions idle in a transaction** (`db.idle_in_transaction`). Selfguard asks for approval to end them. Look at what they hold before approving: `SELECT pid, now() - state_change AS idle_for, left(query, 120) FROM pg_stat_activity WHERE state LIKE 'idle in transaction%';`. Then set `idle_in_transaction_session_timeout` on Govern's role.
- **Lock waits or a long transaction.** Find the blocker: `SELECT pid, pg_blocking_pids(pid), wait_event, left(query, 120) FROM pg_stat_activity WHERE wait_event_type = 'Lock';` and end it with `pg_cancel_backend` if it is not a migration.
- **Tenancy drift while running** (`job.failing` on `tenancy.drift`). The next restart will refuse to start. Fix as in the table above before you restart.

## Mail

Every failed send is logged with `event=email_delivery_failed` and a reason; `GET /api/v1/admin/email/health` shows the last failure and `POST /api/v1/admin/email/probe` tests the configuration.

| Reason | Fix |
|---|---|
| `not_configured` | Set `NI_AUTH_SMTP_HOST` |
| `connect`, `timeout` | Check host, port and firewall; `localhost` needs a local mail server |
| `tls` | STARTTLS is required by default; check `NI_AUTH_SMTP_STARTTLS` and the server certificate |
| `auth` | Check the SMTP credentials |
| `rejected` | The provider refused the sender or recipient (for example an unverified domain or the SES sandbox) |
| `circuit_open` | Sends are paused after repeated failures; fix the cause and the breaker closes by itself |

## Evidence

| Symptom | Cause | Fix |
|---|---|---|
| Every existing audit chain fails verification | `NI_AUTH_AUDIT_HMAC_KEY` changed | Restore the original key |
| Every context evidence chain reported broken (`observability.chain.broken`) | `NI_OBSERVABILITY_SIGNING_KEY` changed, or earlier evidence was signed with the public development key | Restore the original key; evidence signed with the development key cannot be trusted and stays reported |
| Tiering routes answer `503 tiering_unavailable` | Tiering is not configured | Expected without `NI_AUTH_AUDIT_TIERING_BUCKET` |
| The `audit.tiering` job fails and every entry stays hot | The bucket lacks versioning, Object Lock or replication, or a range did not verify | Read the job's error in `/metrics/selfguard`; fix the bucket, or investigate the range before anything moves |
| An export download answers `409 export_integrity` | The stored file is missing or no longer matches its manifest | Check the object store; with several replicas it must be shared |
| `GET /api/v1/audit-logs` lacks old entries | They moved to the cold tier | Use `GET /api/v1/audit-logs/query` or an export |

## Context Platform

| Symptom | Cause | Fix |
|---|---|---|
| Retrieval answers `503 context.misconfigured` | A Context setting is invalid | Read the start log; `GET /api/v1/context/status` shows `configurationValid` |
| Retrieval answers `503 context.authz.unavailable` | A stored deny rule does not parse, or the database failed | `GET /api/v1/context/deny-rules` shows `valid: false` and the parser's message; fix the rule |
| Adding a source answers `503 context.indexer.not_configured` | `KEYSTONE_CONTEXT_CHUNK_MAX_TOKENS=0` | Unset it or set a chunk size |
| A source is in `error` | No GitHub App credentials, the App is not installed on the repository, or GitHub refused the token | `GET /api/v1/context/sources/{id}/status` shows `lastError` and `lastRun`; fix and re-index |
| Saving the GitHub App answers `409 context.github_app.vault_not_ready` | No KV version 2 engine at `secret/` in the organisation's vault | Mount one |
| The feed answers `503 context.events.unavailable` | A `KEYSTONE_CONTEXT_EVENTS_*` value is invalid | Fix it; only the feed is affected |
| Feed streams end after seconds behind a proxy | The proxy buffers or times out the stream | Disable buffering and raise its read timeout above 30 seconds |
| `pgvector` backend answers `503` | The extension is not installed | Install pgvector in your PostgreSQL image, or use the default `sql` backend |

## Before you escalate

Collect: the request id of a failing request, the server log lines around it, `/metrics/selfguard` from the affected node, open selfguard findings, and a diagnostic bundle (`POST /api/v1/admin/selfguard/actions` with `capture_diagnostics`). Record what you did; every selfguard action and approval is also in the audit log.