Backup and restore

What to back up

Keys a restore needs

A database restored without its keys is not usable. Keep these in your secret store, outside the database backup, and include them in every recovery test:

• KMS_MASTER_KEY: seals the audit signing keys and the manifest signing keys; with another value, the server stops.
• NI_AUTH_AUDIT_HMAC_KEY and NI_OBSERVABILITY_SIGNING_KEY: with another value, every existing chain fails verification.
• The encryption keys (NI_AUTH_TOTP_ENCRYPTION_KEY, NI_AUTH_LDAP_ENCRYPTION_KEY, NI_AUTH_SOCIAL_ENCRYPTION_KEY, NI_AUTH_WEBHOOK_ENCRYPTION_KEY, NI_AUTH_VAULT_ENCRYPTION_KEY, NI_AUTH_SECRETS_ENCRYPTION_KEY, NI_AUTH_DLP_TOKEN_ENCRYPTION_KEY): data encrypted under them cannot be read with another key.
• The peppers (NI_AUTH_API_KEY_PEPPER, NI_AUTH_OAUTH2_PEPPER, NI_AUTH_EMAIL_TOKEN_PEPPER): with another value, API keys and credentials must be issued again.
• The three database role passwords and the OIDC signing key.

Back up PostgreSQL

Use the custom format, from a role that can read every table. In Docker Compose the bootstrap superuser (POSTGRES_USER, default ni_auth) can:

docker compose exec -T postgres pg_dump -U ni_auth -d ni_auth \
  --format custom --compress 9 > keystone-$(date +%Y%m%d-%H%M%S).pgdump

Elsewhere:

pg_dump --host "$DB_HOST" --username "$DB_USER" --dbname "$DB_NAME" \
  --format custom --compress 9 --file "/backup/keystone-$(date +%Y%m%d-%H%M%S).pgdump"

The runtime role keystone_app cannot take a complete backup: row-level security hides every tenant row from it without a tenant. Do not give backup credentials to the running server.

Check every backup is readable:

pg_restore --list /backup/keystone-20260907-120000.pgdump > /dev/null && echo OK

For a recovery point objective of minutes, add WAL archiving (wal_level = replica, archive_mode = on, an archive_command that copies to your archive) and test point-in-time recovery.

Restore PostgreSQL

Restore into a separate, empty database first, verify it, and only then cut over. Do not overwrite a production database while you test.

1. Stop the server and ingest planes, so nothing writes during the restore.

2. Create the roles in the target PostgreSQL. Roles are not part of a database dump. Run the bootstrap as the superuser, with the same three passwords:

   KEYSTONE_OWNER_PASSWORD=... KEYSTONE_APP_PASSWORD=... KEYSTONE_PLATFORM_PASSWORD=... \
     psql -v ON_ERROR_STOP=1 -h <host> -U <superuser> -d <database> -f deploy/postgres/bootstrap-roles.sql

3. Restore as the superuser, keeping privileges (the grants to the runtime roles are part of the dump):

   pg_restore --host <host> --username <superuser> --dbname <database> \
     /backup/keystone-20260907-120000.pgdump

4. Run the bootstrap again. It transfers every object not owned by keystone_owner to it, so objects the restore created under the superuser get the right owner.

5. Apply migrations as the owner, in case the backup predates the release you run: ni-auth migrate up with DATABASE_OWNER_URL.

6. Verify isolation as the superuser: SELECT * FROM keystone_tenancy_problems(); must return no rows.

7. Start with the same keys as the backed-up installation, then check /healthz, sign-in and an audit chain verification (The audit ledger).

On Kubernetes, scale the planes to 0 for the restore, and let helm upgrade run the bootstrap and migration hooks afterwards.

Older restore notes pass --no-owner --no-privileges to pg_restore. With database-enforced tenant isolation that drops the runtime roles' grants; keep privileges as above, and let the bootstrap fix ownership.

The built-in scheduled backup

When NI_AUTH_BACKUP_ENCRYPTION_KEY is set, the server runs pg_dump every 6 hours, encrypts the dump with AES-256-GCM and uploads it to an S3 or S3-compatible bucket, keeping 30 days.

• Requests are signed with Signature Version 4, without a cloud SDK.
• A configured backup that cannot start (invalid key, missing region or credentials) stops the server rather than leaving it without the backups it asked for.
• The dump is taken through DATABASE_PLATFORM_URL; pg_dump must be on the server's path.
• The schedule runs as the supervised, critical job backup.scheduler, so a failing schedule raises a selfguard finding.

Limits in this release. The routes to list, verify and restore these backups are not served, so there is no supported command to restore from them. Treat the scheduled backup as an additional copy, keep your own pg_dump or WAL-based backups as the restore path, and keep the encryption key with your other keys. A restore through the server would need DATABASE_OWNER_URL in its environment, which the role separation keeps out of it.

Test restores

Restore into a separate environment at least monthly, from the most recent backup, and record the result: the backup's age, the time the restore took, sign-in, an audit chain verification and a context retrieval. Run the same test after any change to keys or roles.