---
title: "Monitoring and self-healing"
---

# Monitoring and self-healing

Health and readiness probes, Prometheus metrics, and selfguard, Govern's own anomaly detection, admission control and remediation playbooks, with the operator API, kill switch, drills and the rollout gate.

## Probes and metrics

| Endpoint | Authentication | What |
|---|---|---|
| `GET /healthz` | none | The server and its database: `{"status": "ok", "database": "ok"}`. Liveness |
| `GET /readyz` | none | Readiness, with the database pinged (2 s deadline) and `email` status |
| `GET /readyz/selfguard` | none | Fails (`503 selfguard_draining`) only while selfguard drains the node; does not check the database. The Helm chart's readiness probe |
| `GET /metrics` | `NI_AUTH_METRICS_TOKEN` or a super admin | Prometheus metrics for every organisation |
| `GET /metrics/selfguard` | the same | Break-glass status that works while the database is down |
| `GET /metrics/selfguard/rollout-gate` | the same | The rollout gate, for deploy tooling |

`NI_AUTH_METRICS_TOKEN` must be at least 32 bytes (`openssl rand -base64 48`). Scrape with it as a bearer token.

The probes, `/metrics` and `/metrics/selfguard` are exempt from admission control, so they answer while the node is overloaded.

## Selfguard

Selfguard keeps each Govern process healthy on its own. It watches the process and the dependencies Govern needs (PostgreSQL first), detects anomalies, sheds load under pressure, repairs what it safely can, verifies that the repair worked, and escalates to operators when it did not. It is about Govern itself, not tenant workloads.

- **Air-gap first.** No external calls. Alerts go to the log and, optionally, to a webhook you run.
- **Audited.** Every automatic or operator action and every finding transition is written to the audit ledger in the same transaction as the change.
- **Database outages are the main case.** Detection, healing and alerting run from memory; history is written to PostgreSQL in the background, buffered (up to 10,000 records) while it is down.
- **Safe by construction.** Actions run under a mode, a cluster-wide kill switch, cooldowns, an hourly budget (12 per node), prechecks and post-action verification. Disruptive actions need an operator's approval.

It also runs in the data plane (`keystone-bridge`), the compliance plane (`keystone-comply`), the provisioner and the sandbox daemons, from memory and without history or approvals.

### What it watches

| Area | Signals |
|---|---|
| HTTP | Requests, errors, error ratio, p50 and p99 latency, shed requests |
| Go runtime | Goroutines, heap, memory against `GOMEMLIMIT` or the container limit, GC pauses, scheduler latency |
| Process | Open file descriptors against the limit |
| Database pool | In use, idle, queueing time |
| PostgreSQL server | Connection slots in use, longest transaction, sessions idle in a transaction, lock waits, replication lag (one node samples every 30 s) |
| Canaries | The server fetches its own JWKS and OpenID discovery document over loopback every 30 s |
| Disks | Free space of the diagnostics, temporary and data directories, and `NI_AUTH_SELFGUARD_DISK_PATHS` |
| Certificates | Expiry of the configured certificate files and `NI_AUTH_SELFGUARD_CERT_FILES`, re-read hourly |
| Background jobs | Heartbeats, runs, durations, failures, crash loops and stalls of every supervised job |
| Cluster | The same finding on most nodes, lost nodes, silent nodes, and a new release that regresses against the previous one |

Set `GOMEMLIMIT` or a container memory limit; without either, memory pressure cannot be measured.

### Detectors and playbooks

| Rule | Condition | Playbook |
|---|---|---|
| `runtime.memory_pressure` | Memory at 85% of the limit for 3 ticks | capture diagnostics, release memory, shed batch, drain node |
| `process.fd_exhaustion` | 85% of file descriptors for 2 ticks | capture diagnostics, recycle DB connections, shed batch, drain node |
| `db.pool_saturation` | Pool 90% in use and queueing | shed batch, capture diagnostics |
| `db.server_connections` | 90% of PostgreSQL connection slots, 2 samples | recycle DB connections, capture diagnostics |
| `db.idle_in_transaction` | A Govern session idle in a transaction for 10 min | terminate idle transactions (approval) |
| `probe.failing/database` | The database probe failed 3 times | recycle DB connections |
| `http.slo_fast_burn` | Availability budget burning 14.4 times too fast over 1 h and 5 min | capture diagnostics |
| `http.latency_anomaly` | p99 well above the learned baseline for that hour of the week | shed batch, capture diagnostics |
| `job.stalled` | A job missed its heartbeat or overran | restart the job |
| `disk.pressure` | Below 15% and 10 GiB free (warning), or below 1 GiB (critical) | prune diagnostics |
| `cert.expiry` | A certificate expires within 30 days (critical within 7) | operators |

The full list, with thresholds, is in the operator runbook shipped with each release. Baselines warm up first and never fire before that.

### Admission control

The outermost HTTP handler limits concurrency adaptively (starting at 256, between 32 and 4096) and classifies requests:

| Class | Requests | Admitted while in flight is below |
|---|---|---|
| exempt | `/healthz`, `/livez`, `/readyz`, `/readyz/selfguard`, `/metrics`, `/metrics/selfguard` | always |
| critical | `/.well-known/`, token, revoke, userinfo and introspection endpoints, `/api/v1/auth/`, `/api/v1/authz/check`, the selfguard API | 125% of the limit |
| normal | everything else | 100% (90% in protect mode) |
| batch | paths with a segment such as `export`, `exports`, `download`, `report`, `bulk`, `import`, `reindex` | 70% (0% in protect mode) |

Once in-flight requests reach half the limit, one organisation may hold at most a quarter of it (tenant bulkheads). A rejected request gets `503` (node overloaded) or `429` (tenant share exceeded) with `Retry-After` and `X-Keystone-Shed`. Long-lived streams give their slot back once they start. `NI_AUTH_SELFGUARD_ADMISSION=measure` measures without rejecting.

## Operate selfguard

All routes are for super admins. Bodies are JSON; every change needs a `reason` or `note`, recorded in the audit event.

```bash
export KS=https://<your Govern host>
alias ks='curl -sS -H "Authorization: Bearer $ADMIN_TOKEN" -H "Content-Type: application/json"'

ks $KS/api/v1/admin/selfguard/nodes
ks "$KS/api/v1/admin/selfguard/findings?state=open"
ks "$KS/api/v1/admin/selfguard/actions?limit=20"
```

If the API answers `401` during an incident, the database is probably down and sessions cannot be checked. Use the break-glass status with the metrics token:

```bash
curl -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" $KS/metrics/selfguard
```

| Task | Request |
|---|---|
| Acknowledge a finding | `POST /api/v1/admin/selfguard/findings/{id}/acknowledge` `{"note": "..."}` |
| Approve or reject a disruptive action | `POST /api/v1/admin/selfguard/actions/{id}/approve` or `/reject` `{"note": "..."}`, within the approval window (15 min) |
| Pause every automatic action (kill switch) | `PATCH /api/v1/admin/selfguard/settings` `{"actions_paused": true, "paused_reason": "..."}` |
| See what it would do without acting | `PATCH .../settings` `{"mode": "dry_run", "reason": "..."}`; back with `"enforce"` |
| Run an action on a node now | `POST /api/v1/admin/selfguard/actions` `{"node_id", "action", "subject"?, "note"}` |
| Rehearse detection and alerting | `POST /api/v1/admin/selfguard/drills` `{"node_id", "rule", "severity", "duration", "live"?, "note"}` |

The kill switch reaches every node within one tick (10 s). Detection and alerting continue while actions are paused. A drill raises a finding whose message starts with `[DRILL]`; its playbook runs as a dry run unless `"live": true`. Run a drill after enabling selfguard, after changing the alert webhook, and at on-call handover.

Disruptive actions:

- `drain_node` makes `/readyz` answer `503` for 3 minutes, so load balancers route around the node. It is never proposed for the only active node or while another node drains. Approve when the node is clearly sick and the others have headroom.
- `terminate_idle_transactions` ends Govern's own sessions idle in a transaction for over 10 minutes. Before approving, look at what they hold (`pg_stat_activity`); reject when it is a migration or a maintenance session. Then set `idle_in_transaction_session_timeout` on Govern's database role.

Diagnostic bundles (`goroutines.txt`, `heap.pprof`, `status.json`) are written under `NI_AUTH_SELFGUARD_DIAGNOSTICS_DIR`; mount a volume there to keep them across restarts.

### Alerts

Alerts go to the structured log and, when `NI_AUTH_SELFGUARD_ALERT_WEBHOOK_URL` is set, are posted as JSON with kinds `finding.opened`, `finding.escalated`, `finding.resolved`, `action.approval_required`, `action.failed`, `action.unverified` and `remediation.exhausted`. Each request carries `X-Keystone-Timestamp` and `X-Keystone-Signature: sha256=<hex HMAC-SHA256(secret, timestamp + "." + body)>`; reject timestamps older than a few minutes. Delivery is retried up to 5 times.

### The rollout gate

```bash
curl --fail -sS -H "Authorization: Bearer $NI_AUTH_METRICS_TOKEN" \
  "$KS/metrics/selfguard/rollout-gate?release=$TAG&min_soak=10m&strict=true"
```

It answers `{"release", "proceed", "reasons", ...}` and, with `strict=true`, `412` while: no node reports the release; a node running it is not ready or has an open critical finding; it has run for less than `min_soak`; it regresses against the previous release (error ratio more than twice, or p99 more than 1.5 times, the baseline's, with at least 200 requests on each side); or a critical cluster-wide finding is open. Use it as an Argo Rollouts analysis or a Helm post-upgrade hook. Govern does not roll back by itself; the gate lets your tooling do it. The Helm chart sets `NI_AUTH_SELFGUARD_RELEASE` to the image tag.

## Settings

All optional. An invalid value stops the start with a message naming it.

| Variable | Default | Meaning |
|---|---|---|
| `NI_AUTH_SELFGUARD_ENABLED` | `true` | `false` turns detection, healing and shedding off; jobs stay supervised |
| `NI_AUTH_SELFGUARD_MODE` | `enforce` | `observe`, `dry_run` or `enforce`, until an operator stores a mode |
| `NI_AUTH_SELFGUARD_TICK` | `10s` | Detection interval |
| `NI_AUTH_SELFGUARD_MAX_ACTIONS_PER_HOUR` | `12` | Automatic actions per node per hour |
| `NI_AUTH_SELFGUARD_AUTO_APPROVE` | empty | Disruptive actions that run without approval, for example `drain_node` |
| `NI_AUTH_SELFGUARD_APPROVAL_TTL` | `15m` | Approval window |
| `NI_AUTH_SELFGUARD_DIAGNOSTICS_DIR` | `$TMPDIR/keystone-selfguard` | Diagnostic bundles |
| `NI_AUTH_SELFGUARD_DIAGNOSTICS_KEEP` | `10` | Bundles kept |
| `NI_AUTH_SELFGUARD_DRAIN_DURATION` | `3m` | How long a drain reports not ready |
| `NI_AUTH_SELFGUARD_RETENTION_DAYS` | `90` | History retention |
| `NI_AUTH_SELFGUARD_SLO_AVAILABILITY` | `0.999` | Availability objective of the burn-rate rules |
| `NI_AUTH_SELFGUARD_MEMORY_PRESSURE_RATIO` | `0.85` | Memory pressure threshold |
| `NI_AUTH_SELFGUARD_ALERT_WEBHOOK_URL`, `NI_AUTH_SELFGUARD_ALERT_WEBHOOK_SECRET` | empty | Alert webhook and its HMAC secret (at least 32 bytes) |
| `NI_AUTH_SELFGUARD_ADMISSION` | `on` | `on` or `measure` |
| `NI_AUTH_SELFGUARD_ADMISSION_INITIAL_LIMIT`, `_MIN_LIMIT`, `_MAX_LIMIT` | 256, 32, 4096 | Adaptive limit bounds |
| `NI_AUTH_SELFGUARD_ADMISSION_STATIC_LIMIT` | `0` | Above 0 pins the limit |
| `NI_AUTH_SELFGUARD_ADMISSION_TENANT_SHARE` | `0.25` | One organisation's share once bulkheads engage |
| `NI_AUTH_SELFGUARD_RELEASE` | build version | The release the rollout gate compares |
| `NI_AUTH_SELFGUARD_DISK_PATHS` | empty | Extra filesystems to watch, `label=/path` |
| `NI_AUTH_SELFGUARD_CERT_FILES` | empty | Certificate files to watch, `label=/file.pem` |

Give server pods a `terminationGracePeriodSeconds` of at least 30 (the chart uses 60), the bridge at least 50 and the compliance plane at least 45, so their last flushes finish on shutdown.

## Metrics to alert on

| Metric | Alert when |
|---|---|
| `keystone_selfguard_findings_active{rule,severity}` | Any critical finding is active |
| `keystone_selfguard_draining` | A node drains |
| `keystone_selfguard_journal_dropped_total` | It increases: history was dropped during a long database outage |
| `keystone_selfguard_shed_requests_total{priority,reason}` | Normal or critical traffic is shed |
| `keystone_selfguard_breaker_state{breaker}` | A breaker is open (2) |
| `keystone_email_delivery_healthy` | It is 0 for 10 minutes |

## Not covered yet

- No console page for selfguard beyond the platform area's self-healing page; the API is complete.
- No automatic rollback of a bad release: the rollout gate's `412` lets your tooling do it.
- No circuit breaker per model provider in the model gateway yet.