--- title: "Retention and tiering" --- # Retention and tiering How long audit entries stay in PostgreSQL, how older entries move to signed Parquet segments in an Object Lock bucket, and how to configure tiering on your own installation. Without tiering, the whole audit log stays in PostgreSQL. With tiering, entries older than each organisation's **hot window** move to your audit bucket in signed Parquet segments, and stay provable, queryable and exportable. Nothing in Govern deletes an audit object. | Tier | Where | Lock | |---|---|---| | Hot | PostgreSQL | The immutability trigger | | Cold | Audit bucket, Parquet, SSE-KMS, storage class `STANDARD_IA` | None; the signed manifest detects any change | | Archive | Audit bucket, Parquet, SSE-KMS, `GLACIER_IR` | S3 Object Lock in compliance mode until the newest entry of the object plus the archive retention | ## The rules - **Hot window.** On a self-hosted installation, `NI_AUTH_AUDIT_HOT_WINDOW_DAYS` (default 90). Never below 90 days. Platform entries use `NI_AUTH_AUDIT_PLATFORM_HOT_WINDOW_DAYS`. - **Archive retention.** On a self-hosted installation, `NI_AUTH_AUDIT_ARCHIVE_YEARS` (default 0: the cold tier, no Object Lock), at most 10 years. Platform entries use `NI_AUTH_AUDIT_PLATFORM_ARCHIVE_YEARS`. - On the hosted service both come from the account's edition. - **Fail toward keeping.** A hot window that cannot be read keeps the organisation's entries hot. An archive retention that cannot be read postpones sealing. - **Ratchet.** Retention is extended, never shortened: when an organisation's archive retention grows, the Object Lock of its objects is extended; when it shrinks, nothing changes. - **Never deleted.** A row leaves PostgreSQL only after the object holding it was written, read back and matched field by field. The bucket client has no delete operation. - **Legal hold.** An active legal hold on an organisation or a user keeps the organisation's entries hot. ## Segments The job seals the ledger into segments: contiguous id ranges, every organisation included, older than 90 days. A segment is sealed when it reaches `NI_AUTH_AUDIT_TIERING_SEGMENT_ROWS` (default 50,000) entries or its oldest entry is a day past the cutoff. Before sealing, every entry's HMAC and every chain link is verified, including the link to the previous segment; a range that does not verify is never archived, and tiering stops with every entry still hot until an operator looks. Under `/segments/-/`: | File | Content | |---|---| | `manifest.json` | Id range, entry count, first previous hash and last entry hash, the SHA-256 of the previous segment's manifest, the spine and every object (path, SHA-256, size, rows, storage class), encryption, KMS key, cell | | `manifest.sig.json` | Ed25519 signature with the audit signing key over `keystone-audit-cold-segment/v1\n` followed by the manifest bytes | | `spine.parquet` | Id, previous hash, entry hash and a per-row reference of every entry | | `objects/.parquet` | One organisation's entries in their stored, signed form | Object and row references are hashes that change in every segment, so no file names a tenant. Parquet files use a flat schema and GZIP pages; Athena, DuckDB, Spark and pyarrow read them. Each organisation's entries leave PostgreSQL, segment by segment and in order, when the newest entry of its object is older than its hot window. What an organisation has in the cold tier is always a prefix of its ledger. ## Reading both tiers - Audit exports read both tiers, so moving entries never takes them out of an organisation's export of its own records. - `GET /api/v1/audit-logs/query` reads both tiers ([The audit ledger](https://neuralinverse.com/docs/evidence/audit-ledger/)). - `GET /api/v1/audit-logs` (the console list) reads the hot tier only. - `GET /api/v1/audit-logs/tiering` (`audit:read`) shows the organisation's hot window, archive retention, tier and cold tier summary. - `GET /api/v1/audit-logs/cold-segments` lists the organisation's sealed objects with their manifest digest and signing key. The tiering routes answer `503 tiering_unavailable` when tiering is not configured. ## Turn tiering on ### The bucket - Versioning and Object Lock enabled (required for archive objects). - SSE-KMS, with the key named on every write; the store refuses an object it did not encrypt with SSE-KMS. - Cross-region replication enabled, unless you set `NI_AUTH_AUDIT_TIERING_REQUIRE_REPLICATION=false`. - Block Public Access. Before every run the job reads the bucket's Object Lock, versioning and replication configuration; a bucket that is not ready stops the run before anything moves. Requests are signed with Signature Version 4 without a cloud SDK, so an S3-compatible store with Object Lock (for example MinIO) works on premises and air-gapped. The server's identity needs only: `s3:PutObject`, `s3:GetObject`, `s3:PutObjectRetention`, `s3:GetBucketObjectLockConfiguration`, `s3:GetBucketVersioning`, `s3:GetReplicationConfiguration`, and `kms:GenerateDataKey` and `kms:Decrypt` on the key. Not `s3:DeleteObject`, not `s3:BypassGovernanceRetention`. ### Settings | Variable | Default | | |---|---|---| | `NI_AUTH_AUDIT_TIERING_BUCKET` | | Turns tiering on | | `NI_AUTH_AUDIT_TIERING_KMS_KEY_ID` | | Required: the SSE-KMS key | | `NI_AUTH_AUDIT_TIERING_REGION` | `AWS_REGION` | Bucket region | | `NI_AUTH_AUDIT_TIERING_ENDPOINT` | AWS S3 in the region | S3-compatible endpoint | | `NI_AUTH_AUDIT_TIERING_PATH_STYLE` | `false` | Path-style addressing | | `NI_AUTH_AUDIT_TIERING_ACCESS_KEY_ID`, `NI_AUTH_AUDIT_TIERING_SECRET_ACCESS_KEY` | | Else the `AWS_*` variables, then the instance role | | `NI_AUTH_AUDIT_TIERING_PREFIX` | `audit/` | Object key prefix | | `NI_AUTH_AUDIT_TIERING_REQUIRE_REPLICATION` | `true` | Refuse to run without cross-region replication | | `NI_AUTH_AUDIT_TIERING_INTERVAL` | `1h` | How often the job runs | | `NI_AUTH_AUDIT_TIERING_SEGMENT_ROWS` | `50000` | Most entries in a segment | | `NI_AUTH_AUDIT_HOT_WINDOW_DAYS` | `90` | Hot window | | `NI_AUTH_AUDIT_PLATFORM_HOT_WINDOW_DAYS` | the above | Platform entries | | `NI_AUTH_AUDIT_ARCHIVE_YEARS` | `0` | Archive retention | | `NI_AUTH_AUDIT_PLATFORM_ARCHIVE_YEARS` | `0` | Platform entries | An invalid value stops the server. The server also refuses to start when the ledger already has entries in the cold tier and no bucket is configured (exports and queries would leave those entries out), and when there is no `KMS_MASTER_KEY` (manifests are signed with the persisted key set only). ## Operations The job runs as the supervised job `audit.tiering`, one run at a time across replicas. Each run is recorded with the segments sealed, objects and rows evicted, organisations kept hot, and any error. A failed run (bucket not ready, unreadable retention, a range that does not verify, an object that does not match) leaves every entry hot and fails the job, so [selfguard](https://neuralinverse.com/docs/deploy/monitoring/) reports it; the next run retries. A crash while a segment is written leaves it pending, and the next run finishes the same segment. Seals are audited as `audit.tiering.segment_sealed`, evictions as `audit.tiering.evicted` in the organisation's own log. To prove the result, use [`keystone-audit-verify cold`](https://neuralinverse.com/docs/evidence/verify-offline/).