--- title: "Upgrades and migrations" --- # Upgrades and migrations Upgrade Neural Inverse Govern safely, run database migrations as the schema owner, handle releases that need special steps, and recover from a failed or dirty migration. ## The short version 1. Read the release notes for steps the release requires before it starts. 2. Back up the database ([Backup and restore](https://neuralinverse.com/docs/deploy/backup-restore/)). 3. Run the role bootstrap and the migrations as the schema owner, `keystone_owner`. 4. Start the new version, and check `/healthz` and `/readyz`. Docker Compose and the Helm chart do step 3 for you. Take a backup and restore it rather than migrating down in production. ## Which database role runs migrations Migrations always run as **`keystone_owner`**, through `DATABASE_OWNER_URL`. Never run them through `DATABASE_URL`. | Variable | Role | Used for | |---|---|---| | `DATABASE_OWNER_URL` | `keystone_owner`: owns the schema, `BYPASSRLS`, no superuser | Migrations and restores only. Never given to the running server | | `DATABASE_URL` | `keystone_app`: no superuser, no `BYPASSRLS`, no DDL | The server and the ingest plane at runtime, under row-level security | | `DATABASE_PLATFORM_URL` | `keystone_platform`: `BYPASSRLS`, no DDL | The named cross-organisation tasks; `keystone-audit-verify export` and `cell init` | `DATABASE_URL` cannot run migrations: the runtime role has no DDL rights by design, and the server refuses to start on a role that could (a superuser, a `BYPASSRLS` role, or one that owns objects). The server's own `migrate` command reads only `DATABASE_OWNER_URL` and stops with "DATABASE_OWNER_URL is required" without it. > Older upgrade notes show `migrate -database "$DATABASE_URL" up`. That predates the separation of roles and is wrong for current releases: use `DATABASE_OWNER_URL`. ## Docker Compose ```bash docker compose pull # or build the new release docker compose up -d ``` On every `up`, the `db-roles` service runs `deploy/postgres/bootstrap-roles.sql` as the PostgreSQL superuser (idempotent), then the `migrate` service applies pending migrations as `keystone_owner` and exits `0`. The server starts only after both completed. To run the migrations alone: ```bash docker compose run --rm migrate ``` ## Kubernetes `helm upgrade` runs pre-upgrade hooks in order: the role bootstrap (`databaseRoles.bootstrap`), the migrations as `keystone_owner` from the `keystone-migrate` image of the same release, then `ni-auth cell init`. Only the hooks mount the owner's credentials (`secrets.databaseAdmin`). Database migrations are not rolled back by `helm rollback`. ## Anything else ```bash # as a PostgreSQL superuser, with the three role passwords in the environment KEYSTONE_OWNER_PASSWORD=... KEYSTONE_APP_PASSWORD=... KEYSTONE_PLATFORM_PASSWORD=... \ psql -v ON_ERROR_STOP=1 -h -U -d -f deploy/postgres/bootstrap-roles.sql # as the schema owner DATABASE_OWNER_URL=postgres://keystone_owner:...@/ ni-auth migrate up ``` | Command | What | |---|---| | `ni-auth migrate up` | Apply all pending migrations | | `ni-auth migrate down [n]` | Roll back `n` migrations (default 1; `-all` for everything) | | `ni-auth migrate status` | Current version (`version` is an alias) | | `ni-auth migrate lint` | Check every migration for safety rules; needs no database | The migration runner records who ran it from `KEYSTONE_MIGRATE_USER` (default `keystone-cli`). ## After the migrations The server checks at start, and refuses to start when any check fails: - `DATABASE_URL` is not a superuser, has no `BYPASSRLS`, owns nothing and cannot create roles or databases. - `DATABASE_PLATFORM_URL` is set, differs from `DATABASE_URL`, has `BYPASSRLS`, is not a superuser and owns nothing. - `keystone_tenancy_problems()` returns no rows: every tenant table is isolated as its decision says. - Outside development, neither URL uses a password shipped in the repository. While it runs, the server repeats the tenancy check every five minutes (the job `tenancy.drift`); a problem raises a critical selfguard finding before the next restart would refuse to start. Verify by hand if you want: ```sql -- as the superuser: must return no rows SELECT * FROM keystone_tenancy_problems(); ``` ## Releases with special steps Read each release's notes. Examples of steps releases have required: - **Database-enforced tenant isolation.** The first release with the three roles needs a short maintenance window on an existing installation: stop the planes, run the role bootstrap (it transfers ownership of every object to `keystone_owner`), migrate as the owner, verify `keystone_tenancy_problems()`, switch `DATABASE_URL` to `keystone_app`, add `DATABASE_PLATFORM_URL`, remove the superuser's credentials from the planes, start, and watch the logs for `no tenant for this statement` (SQLSTATE `42501`), which marks a code path without a tenant: report it, do not grant more. - **No rolling upgrade.** A release that changes how stored values are encrypted may require every old replica to stop before the new one starts. On Kubernetes, scale the server to 0, wait until no old pod runs, then `helm upgrade`; with autoscaling, scale back to the minimum afterwards. - **Per-organisation audit chains.** Run the migrations before starting the new servers; old servers fail their audited requests from the switch until they are replaced (fail closed). That release's down migration leaves entries written after the switch unverifiable, so restore a backup instead of migrating down. - **New required keys.** Releases have made `NI_OBSERVABILITY_SIGNING_KEY` and `KMS_MASTER_KEY` required. The server refuses to start without them, and keeping them stable matters more than the upgrade (see [Deploy and operate](https://neuralinverse.com/docs/deploy/)). - **PostgreSQL 15 or later.** Migrations stop with "Keystone requires PostgreSQL 15 or later" on an older server, before changing anything. Upgrade PostgreSQL first. Some migrations build indexes concurrently or validate constraints without the runner's statement timeout, so they can take a while on a large audit ledger without blocking writes. A follow-up migration checks the index is valid and stops the upgrade (dirty) if a build was interrupted; that migration's file has the repair steps. ### Rolling out with the selfguard gate Between rollout steps, ask the rollout gate whether the new release is healthy on live traffic: ```bash curl --fail -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" \ "https:///metrics/selfguard/rollout-gate?release=$TAG&min_soak=10m&strict=true" ``` With `strict=true` it answers `412` while the release has not soaked, a node running it is not ready or has a critical finding, or the release regresses against the previous one. See [Monitoring and self-healing](https://neuralinverse.com/docs/deploy/monitoring/). ## When something goes wrong ### Roll back 1. Stop the new version. 2. Restore the backup you took before the upgrade (preferred), or migrate down as the owner the number of migrations the upgrade added: `ni-auth migrate down `. Data migrations that delete data cannot be rolled back. 3. Start the old version. ### Dirty migration state If a migration was interrupted or failed, the database is marked dirty at that version and the runner refuses to continue ("database is dirty at version N — manual intervention required"). Read the failed migration's file for repair steps (several have them) and fix the cause. Then, as the owner, clear the flag to the last good version and apply the failed migration again with the migration tool (the Docker Compose `migrate` service uses `migrate/migrate`): ```bash migrate -path db/migrations -database "$DATABASE_OWNER_URL" force migrate -path db/migrations -database "$DATABASE_OWNER_URL" up 1 ``` Run the role bootstrap again first if the failure was a privilege error.