---
title: "Backup and restore"
---

# Backup and restore

What to back up in a Govern installation, how to back up and restore PostgreSQL with the three database roles intact, the keys you must keep, and what the built-in scheduled backup does and does not do.

## What to back up

| Data | Where | How |
|---|---|---|
| Everything Govern stores: identities, policies, the audit ledger's hot tier, Context spaces and evidence | PostgreSQL | `pg_dump`, or WAL archiving for point-in-time recovery |
| Exports, artifacts, agent release binaries | The shared object store: a bucket (`NI_OBJECT_STORE_S3_*`) or a directory (`NI_OBJECT_STORE_DIR`, the `keystone_data` volume in Docker Compose) | Bucket replication or a volume snapshot |
| The audit cold and archive tiers | The audit bucket | Its cross-region replication; nothing in Govern deletes from it |
| Telemetry cold files, if the pipeline is on | `NI_TELEMETRY_COLD_S3_BUCKET` or `NI_TELEMETRY_COLD_DIR` | Bucket replication or a volume snapshot |
| Keys | Your secret store | Kept outside the backup, with the same care |

### Keys a restore needs

A database restored without its keys is not usable. Keep these in your secret store, outside the database backup, and include them in every recovery test:

- `KMS_MASTER_KEY`: seals the audit signing keys and the manifest signing keys; with another value, the server stops.
- `NI_AUTH_AUDIT_HMAC_KEY` and `NI_OBSERVABILITY_SIGNING_KEY`: with another value, every existing chain fails verification.
- The encryption keys (`NI_AUTH_TOTP_ENCRYPTION_KEY`, `NI_AUTH_LDAP_ENCRYPTION_KEY`, `NI_AUTH_SOCIAL_ENCRYPTION_KEY`, `NI_AUTH_WEBHOOK_ENCRYPTION_KEY`, `NI_AUTH_VAULT_ENCRYPTION_KEY`, `NI_AUTH_SECRETS_ENCRYPTION_KEY`, `NI_AUTH_DLP_TOKEN_ENCRYPTION_KEY`): data encrypted under them cannot be read with another key.
- The peppers (`NI_AUTH_API_KEY_PEPPER`, `NI_AUTH_OAUTH2_PEPPER`, `NI_AUTH_EMAIL_TOKEN_PEPPER`): with another value, API keys and credentials must be issued again.
- The three database role passwords and the OIDC signing key.

## Back up PostgreSQL

Use the custom format, from a role that can read every table. In Docker Compose the bootstrap superuser (`POSTGRES_USER`, default `ni_auth`) can:

```bash
docker compose exec -T postgres pg_dump -U ni_auth -d ni_auth \
  --format custom --compress 9 > keystone-$(date +%Y%m%d-%H%M%S).pgdump
```

Elsewhere:

```bash
pg_dump --host "$DB_HOST" --username "$DB_USER" --dbname "$DB_NAME" \
  --format custom --compress 9 --file "/backup/keystone-$(date +%Y%m%d-%H%M%S).pgdump"
```

The runtime role `keystone_app` cannot take a complete backup: row-level security hides every tenant row from it without a tenant. Do not give backup credentials to the running server.

Check every backup is readable:

```bash
pg_restore --list /backup/keystone-20260907-120000.pgdump > /dev/null && echo OK
```

For a recovery point objective of minutes, add WAL archiving (`wal_level = replica`, `archive_mode = on`, an `archive_command` that copies to your archive) and test point-in-time recovery.

## Restore PostgreSQL

Restore into a separate, empty database first, verify it, and only then cut over. Do not overwrite a production database while you test.

1. **Stop the server and ingest planes**, so nothing writes during the restore.
2. **Create the roles** in the target PostgreSQL. Roles are not part of a database dump. Run the bootstrap as the superuser, with the same three passwords:

   ```bash
   KEYSTONE_OWNER_PASSWORD=... KEYSTONE_APP_PASSWORD=... KEYSTONE_PLATFORM_PASSWORD=... \
     psql -v ON_ERROR_STOP=1 -h <host> -U <superuser> -d <database> -f deploy/postgres/bootstrap-roles.sql
   ```

3. **Restore** as the superuser, keeping privileges (the grants to the runtime roles are part of the dump):

   ```bash
   pg_restore --host <host> --username <superuser> --dbname <database> \
     /backup/keystone-20260907-120000.pgdump
   ```

4. **Run the bootstrap again.** It transfers every object not owned by `keystone_owner` to it, so objects the restore created under the superuser get the right owner.
5. **Apply migrations** as the owner, in case the backup predates the release you run: `ni-auth migrate up` with `DATABASE_OWNER_URL`.
6. **Verify isolation** as the superuser: `SELECT * FROM keystone_tenancy_problems();` must return no rows.
7. **Start** with the same keys as the backed-up installation, then check `/healthz`, sign-in and an audit chain verification ([The audit ledger](https://neuralinverse.com/docs/evidence/audit-ledger/)).

On Kubernetes, scale the planes to 0 for the restore, and let `helm upgrade` run the bootstrap and migration hooks afterwards.

Older restore notes pass `--no-owner --no-privileges` to `pg_restore`. With database-enforced tenant isolation that drops the runtime roles' grants; keep privileges as above, and let the bootstrap fix ownership.

## The built-in scheduled backup

When `NI_AUTH_BACKUP_ENCRYPTION_KEY` is set, the server runs `pg_dump` every 6 hours, encrypts the dump with AES-256-GCM and uploads it to an S3 or S3-compatible bucket, keeping 30 days.

| Variable | Default | Meaning |
|---|---|---|
| `NI_AUTH_BACKUP_ENCRYPTION_KEY` | unset (off) | 64 hex characters (32 bytes); turns the schedule on |
| `NI_AUTH_BACKUP_S3_BUCKET` | `keystone-backups` | Bucket |
| `NI_AUTH_BACKUP_S3_PREFIX` | `backups` | Key prefix |
| `NI_AUTH_BACKUP_S3_REGION` | `AWS_REGION`, then `AWS_DEFAULT_REGION` | Required; use `us-east-1` for MinIO and most S3-compatible stores |
| `NI_AUTH_BACKUP_S3_ENDPOINT` | Amazon S3 | S3-compatible endpoint |
| `NI_AUTH_BACKUP_S3_ACCESS_KEY`, `NI_AUTH_BACKUP_S3_SECRET_KEY` | | Else the `AWS_*` variables, else the instance role |

- Requests are signed with Signature Version 4, without a cloud SDK.
- A configured backup that cannot start (invalid key, missing region or credentials) stops the server rather than leaving it without the backups it asked for.
- The dump is taken through `DATABASE_PLATFORM_URL`; `pg_dump` must be on the server's path.
- The schedule runs as the supervised, critical job `backup.scheduler`, so a failing schedule raises a selfguard finding.

**Limits in this release.** The routes to list, verify and restore these backups are not served, so there is no supported command to restore from them. Treat the scheduled backup as an additional copy, keep your own `pg_dump` or WAL-based backups as the restore path, and keep the encryption key with your other keys. A restore through the server would need `DATABASE_OWNER_URL` in its environment, which the role separation keeps out of it.

## Test restores

Restore into a separate environment at least monthly, from the most recent backup, and record the result: the backup's age, the time the restore took, sign-in, an audit chain verification and a context retrieval. Run the same test after any change to keys or roles.