Without tiering, the whole audit log stays in PostgreSQL. With tiering, entries older than each organisation's hot window move to your audit bucket in signed Parquet segments, and stay provable, queryable and exportable. Nothing in Govern deletes an audit object.
| Tier | Where | Lock |
|---|---|---|
| Hot | PostgreSQL | The immutability trigger |
| Cold | Audit bucket, Parquet, SSE-KMS, storage class STANDARD_IA | None; the signed manifest detects any change |
| Archive | Audit bucket, Parquet, SSE-KMS, GLACIER_IR | S3 Object Lock in compliance mode until the newest entry of the object plus the archive retention |
The rules
- Hot window. On a self-hosted installation,
NI_AUTH_AUDIT_HOT_WINDOW_DAYS(default 90). Never below 90 days. Platform entries useNI_AUTH_AUDIT_PLATFORM_HOT_WINDOW_DAYS. - Archive retention. On a self-hosted installation,
NI_AUTH_AUDIT_ARCHIVE_YEARS(default 0: the cold tier, no Object Lock), at most 10 years. Platform entries useNI_AUTH_AUDIT_PLATFORM_ARCHIVE_YEARS. - On the hosted service both come from the account's edition.
- Fail toward keeping. A hot window that cannot be read keeps the organisation's entries hot. An archive retention that cannot be read postpones sealing.
- Ratchet. Retention is extended, never shortened: when an organisation's archive retention grows, the Object Lock of its objects is extended; when it shrinks, nothing changes.
- Never deleted. A row leaves PostgreSQL only after the object holding it was written, read back and matched field by field. The bucket client has no delete operation.
- Legal hold. An active legal hold on an organisation or a user keeps the organisation's entries hot.
Segments
The job seals the ledger into segments: contiguous id ranges, every organisation included, older than 90 days. A segment is sealed when it reaches NI_AUTH_AUDIT_TIERING_SEGMENT_ROWS (default 50,000) entries or its oldest entry is a day past the cutoff. Before sealing, every entry's HMAC and every chain link is verified, including the link to the previous segment; a range that does not verify is never archived, and tiering stops with every entry still hot until an operator looks.
Under <prefix>/segments/<seq>-<segment id>/:
| File | Content |
|---|---|
manifest.json | Id range, entry count, first previous hash and last entry hash, the SHA-256 of the previous segment's manifest, the spine and every object (path, SHA-256, size, rows, storage class), encryption, KMS key, cell |
manifest.sig.json | Ed25519 signature with the audit signing key over keystone-audit-cold-segment/v1\n followed by the manifest bytes |
spine.parquet | Id, previous hash, entry hash and a per-row reference of every entry |
objects/<org_ref>.parquet | One organisation's entries in their stored, signed form |
Object and row references are hashes that change in every segment, so no file names a tenant. Parquet files use a flat schema and GZIP pages; Athena, DuckDB, Spark and pyarrow read them.
Each organisation's entries leave PostgreSQL, segment by segment and in order, when the newest entry of its object is older than its hot window. What an organisation has in the cold tier is always a prefix of its ledger.
Reading both tiers
- Audit exports read both tiers, so moving entries never takes them out of an organisation's export of its own records.
GET /api/v1/audit-logs/queryreads both tiers (The audit ledger).GET /api/v1/audit-logs(the console list) reads the hot tier only.GET /api/v1/audit-logs/tiering(audit:read) shows the organisation's hot window, archive retention, tier and cold tier summary.GET /api/v1/audit-logs/cold-segmentslists the organisation's sealed objects with their manifest digest and signing key.
The tiering routes answer 503 tiering_unavailable when tiering is not configured.
Turn tiering on
The bucket
- Versioning and Object Lock enabled (required for archive objects).
- SSE-KMS, with the key named on every write; the store refuses an object it did not encrypt with SSE-KMS.
- Cross-region replication enabled, unless you set
NI_AUTH_AUDIT_TIERING_REQUIRE_REPLICATION=false. - Block Public Access.
Before every run the job reads the bucket's Object Lock, versioning and replication configuration; a bucket that is not ready stops the run before anything moves. Requests are signed with Signature Version 4 without a cloud SDK, so an S3-compatible store with Object Lock (for example MinIO) works on premises and air-gapped.
The server's identity needs only: s3:PutObject, s3:GetObject, s3:PutObjectRetention, s3:GetBucketObjectLockConfiguration, s3:GetBucketVersioning, s3:GetReplicationConfiguration, and kms:GenerateDataKey and kms:Decrypt on the key. Not s3:DeleteObject, not s3:BypassGovernanceRetention.
Settings
| Variable | Default | |
|---|---|---|
NI_AUTH_AUDIT_TIERING_BUCKET | Turns tiering on | |
NI_AUTH_AUDIT_TIERING_KMS_KEY_ID | Required: the SSE-KMS key | |
NI_AUTH_AUDIT_TIERING_REGION | AWS_REGION | Bucket region |
NI_AUTH_AUDIT_TIERING_ENDPOINT | AWS S3 in the region | S3-compatible endpoint |
NI_AUTH_AUDIT_TIERING_PATH_STYLE | false | Path-style addressing |
NI_AUTH_AUDIT_TIERING_ACCESS_KEY_ID, NI_AUTH_AUDIT_TIERING_SECRET_ACCESS_KEY | Else the AWS_* variables, then the instance role | |
NI_AUTH_AUDIT_TIERING_PREFIX | audit/<cell> | Object key prefix |
NI_AUTH_AUDIT_TIERING_REQUIRE_REPLICATION | true | Refuse to run without cross-region replication |
NI_AUTH_AUDIT_TIERING_INTERVAL | 1h | How often the job runs |
NI_AUTH_AUDIT_TIERING_SEGMENT_ROWS | 50000 | Most entries in a segment |
NI_AUTH_AUDIT_HOT_WINDOW_DAYS | 90 | Hot window |
NI_AUTH_AUDIT_PLATFORM_HOT_WINDOW_DAYS | the above | Platform entries |
NI_AUTH_AUDIT_ARCHIVE_YEARS | 0 | Archive retention |
NI_AUTH_AUDIT_PLATFORM_ARCHIVE_YEARS | 0 | Platform entries |
An invalid value stops the server. The server also refuses to start when the ledger already has entries in the cold tier and no bucket is configured (exports and queries would leave those entries out), and when there is no KMS_MASTER_KEY (manifests are signed with the persisted key set only).
Operations
The job runs as the supervised job audit.tiering, one run at a time across replicas. Each run is recorded with the segments sealed, objects and rows evicted, organisations kept hot, and any error. A failed run (bucket not ready, unreadable retention, a range that does not verify, an object that does not match) leaves every entry hot and fails the job, so selfguard reports it; the next run retries. A crash while a segment is written leaves it pending, and the next run finishes the same segment. Seals are audited as audit.tiering.segment_sealed, evictions as audit.tiering.evicted in the organisation's own log.
To prove the result, use keystone-audit-verify cold.