Start with three questions: does the process run, can it reach PostgreSQL, and what do the logs and selfguard say?
# Docker Compose
docker compose ps
docker compose logs server --since 10m | grep -iE "error|panic|fatal|refused"
curl -s http://localhost:8080/healthz
# Kubernetes
kubectl get pods
kubectl logs <server pod> --since=10m | grep -iE "error|panic|fatal|refused"
# Selfguard, even while the database is down
curl -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" https://<your Govern host>/metrics/selfguard
The server does not start
| Symptom | Cause | Fix |
|---|
| Compose refuses to start and names a variable | A required variable is empty | Set it; generation commands are in Docker Compose |
| The server exits naming a key | A key is missing, too short, all zero bytes or a placeholder (change-me, replace-with, ...) | Generate a real value. NI_AUTH_DEPLOY_ENV=development accepts placeholders for local development only |
DATABASE_OWNER_URL is required from migrate | Migrations run only as keystone_owner | Set DATABASE_OWNER_URL; see Upgrades and migrations |
| The server refuses the runtime role | DATABASE_URL is a superuser, has BYPASSRLS, owns objects, or equals DATABASE_PLATFORM_URL | Point DATABASE_URL at keystone_app and DATABASE_PLATFORM_URL at keystone_platform; run the role bootstrap |
| The server refuses to start because tables are not isolated | keystone_tenancy_problems() returns rows: a migration did not finish, a grant was made by hand, or a partition lacks its parent's privileges | Run the role bootstrap and the migrations as the owner; as the superuser, read SELECT * FROM keystone_tenancy_problems();; revoke hand-made grants. Never grant a runtime role more to make it start |
| Refused for a repository password | DATABASE_URL or DATABASE_PLATFORM_URL uses a development password shipped in the repository | Set real role passwords and run the bootstrap |
| Stops when a KMS-sealed key cannot be opened | KMS_MASTER_KEY differs from the one the keys were sealed under | Restore the original KMS_MASTER_KEY |
| Migrations stop: "Keystone requires PostgreSQL 15 or later" | PostgreSQL 14 or older | Upgrade PostgreSQL first; nothing was changed |
| "database is dirty at version N" | A migration was interrupted | See the dirty migration steps in Upgrades and migrations |
| Stops: audit tiering bucket missing | The ledger already has entries in the cold tier | Configure NI_AUTH_AUDIT_TIERING_BUCKET again |
| Stops: backup cannot start | NI_AUTH_BACKUP_ENCRYPTION_KEY set, but the key, region or credentials are invalid | Fix them, or unset the key |
| Stops on a client IP setting | NI_AUTH_CLIENT_IP_HEADER set without NI_AUTH_TRUSTED_CDN_PROXIES, or a catch-all range | See Client IP addresses |
It runs, but requests fail
| Symptom | Cause | Fix |
|---|
Sign-in returns 500 | No OIDC private key | Set NI_AUTH_OIDC_PRIVATE_KEY_FILE or NI_AUTH_OIDC_PRIVATE_KEY_PEM |
| Tokens fail validation | NI_AUTH_JWT_ISSUER does not equal NI_AUTH_PUBLIC_URL exactly | Make them identical |
Logs show no tenant for this statement (SQLSTATE 42501) | A code path ran without a tenant and failed closed | Report it with the request id; do not grant the runtime role more |
503 with X-Keystone-Shed: node_overloaded, or 429 with tenant_share_exceeded | Admission control is shedding load | Scale out; check that the shed traffic is batch (keystone_selfguard_shed_requests_total{priority}); follow the findings in Monitoring |
remaining connection slots are reserved or too many clients already | PostgreSQL is out of connections | Find who holds them: SELECT usename, application_name, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC;. Lower NI_AUTH_DB_MAX_OPEN across replicas, or put PgBouncer (session mode) in front |
| Rate limits and lockouts hit every user at once | All requests appear to come from the proxy | Set NI_AUTH_TRUSTED_PROXIES (and CDN settings) |
| Every request in the audit log shows the console host | The console's signed client context is not verified | Set NI_AUTH_CONSOLE_CONTEXT_KEYS |
Database incidents
- PostgreSQL unreachable. Selfguard raises
probe.failing/database and recycles connections; /readyz turns not ready. Its history is buffered in memory and written in order when the database returns. Check that PostgreSQL runs and accepts the runtime role's credentials.
- Sessions idle in a transaction (
db.idle_in_transaction). Selfguard asks for approval to end them. Look at what they hold before approving: SELECT pid, now() - state_change AS idle_for, left(query, 120) FROM pg_stat_activity WHERE state LIKE 'idle in transaction%';. Then set idle_in_transaction_session_timeout on Govern's role.
- Lock waits or a long transaction. Find the blocker:
SELECT pid, pg_blocking_pids(pid), wait_event, left(query, 120) FROM pg_stat_activity WHERE wait_event_type = 'Lock'; and end it with pg_cancel_backend if it is not a migration.
- Tenancy drift while running (
job.failing on tenancy.drift). The next restart will refuse to start. Fix as in the table above before you restart.
Mail
Every failed send is logged with event=email_delivery_failed and a reason; GET /api/v1/admin/email/health shows the last failure and POST /api/v1/admin/email/probe tests the configuration.
| Reason | Fix |
|---|
not_configured | Set NI_AUTH_SMTP_HOST |
connect, timeout | Check host, port and firewall; localhost needs a local mail server |
tls | STARTTLS is required by default; check NI_AUTH_SMTP_STARTTLS and the server certificate |
auth | Check the SMTP credentials |
rejected | The provider refused the sender or recipient (for example an unverified domain or the SES sandbox) |
circuit_open | Sends are paused after repeated failures; fix the cause and the breaker closes by itself |
Evidence
| Symptom | Cause | Fix |
|---|
| Every existing audit chain fails verification | NI_AUTH_AUDIT_HMAC_KEY changed | Restore the original key |
Every context evidence chain reported broken (observability.chain.broken) | NI_OBSERVABILITY_SIGNING_KEY changed, or earlier evidence was signed with the public development key | Restore the original key; evidence signed with the development key cannot be trusted and stays reported |
Tiering routes answer 503 tiering_unavailable | Tiering is not configured | Expected without NI_AUTH_AUDIT_TIERING_BUCKET |
The audit.tiering job fails and every entry stays hot | The bucket lacks versioning, Object Lock or replication, or a range did not verify | Read the job's error in /metrics/selfguard; fix the bucket, or investigate the range before anything moves |
An export download answers 409 export_integrity | The stored file is missing or no longer matches its manifest | Check the object store; with several replicas it must be shared |
GET /api/v1/audit-logs lacks old entries | They moved to the cold tier | Use GET /api/v1/audit-logs/query or an export |
Context Platform
| Symptom | Cause | Fix |
|---|
Retrieval answers 503 context.misconfigured | A Context setting is invalid | Read the start log; GET /api/v1/context/status shows configurationValid |
Retrieval answers 503 context.authz.unavailable | A stored deny rule does not parse, or the database failed | GET /api/v1/context/deny-rules shows valid: false and the parser's message; fix the rule |
Adding a source answers 503 context.indexer.not_configured | KEYSTONE_CONTEXT_CHUNK_MAX_TOKENS=0 | Unset it or set a chunk size |
A source is in error | No GitHub App credentials, the App is not installed on the repository, or GitHub refused the token | GET /api/v1/context/sources/{id}/status shows lastError and lastRun; fix and re-index |
Saving the GitHub App answers 409 context.github_app.vault_not_ready | No KV version 2 engine at secret/ in the organisation's vault | Mount one |
The feed answers 503 context.events.unavailable | A KEYSTONE_CONTEXT_EVENTS_* value is invalid | Fix it; only the feed is affected |
| Feed streams end after seconds behind a proxy | The proxy buffers or times out the stream | Disable buffering and raise its read timeout above 30 seconds |
pgvector backend answers 503 | The extension is not installed | Install pgvector in your PostgreSQL image, or use the default sql backend |
Before you escalate
Collect: the request id of a failing request, the server log lines around it, /metrics/selfguard from the affected node, open selfguard findings, and a diagnostic bundle (POST /api/v1/admin/selfguard/actions with capture_diagnostics). Record what you did; every selfguard action and approval is also in the audit log.