---
title: "Telemetry pipeline"
---

# Telemetry pipeline

Run Govern's built-in event stream for agent and service telemetry, from the bridge write-ahead log through hourly PostgreSQL partitions to sealed Parquet files, with no Kafka or external store.

The telemetry pipeline is Govern's own event stream and telemetry store for agent and service events. Its only stateful dependencies are PostgreSQL and object storage: an S3 or S3-compatible bucket, or a local directory for self-hosted and air-gapped installations. Kafka is not needed.

It is **off by default**. With it off, nothing changes.

## How an event travels

1. A producer posts events to the bridge: `POST /api/v1/telemetry/events`, with a device token or a service token. The organisation comes from the credential.
2. The bridge (`keystone-bridge`) validates each event and appends it to a write-ahead log on its own volume. It answers `202` only after the write is on disk (fsync).
3. The bridge delivers the log to the ingest replicas (`keystone-ingest`), each organisation always to the same replica.
4. Ingest writes the events into hourly PostgreSQL partitions, dropping duplicates by event id. The bridge acknowledges only when every part of a batch is stored.
5. A compactor turns each closed hour into one sealed Parquet file per organisation (zstd, SHA-256 and HMAC-SHA256 under `NI_AUTH_AUDIT_HMAC_KEY`), reads it back and verifies it before it records it.
6. Compacted hours older than the hot retention (`NI_TELEMETRY_HOT_RETENTION`, default 168 hours) are dropped from PostgreSQL. Cold files are kept for the organisation's cold retention.

Delivery is at least once and duplicates are dropped at ingest, so no accepted event is lost or stored twice.

### Reading telemetry

Events are searched in the Observe log group `/ni/telemetry` (`FilterLogEvents`, `GetLogEvents`, `CreateExportTask`), which reads both tiers and never reads an hour twice or skips one. Every string passes the account's DLP redaction. The console page is Observe > Telemetry, with status (Healthy, Delayed, Not receiving with the server's reason), the hot window, the archive per day and recent sources.

## Turn it on with Helm

```yaml
telemetryPipeline:
  enabled: true                  # also turns on bridge.stream
  hotRetention: 168h
  cold:
    local:
      create:
        enabled: true            # a ReadWriteMany claim <release>-telemetry-cold, kept on uninstall
        size: 500Gi
bridge:
  stream:
    persistence:
      size: 50Gi
```

Exactly one cold store is required; the chart refuses none or both:

- **A directory** (self-hosted, air-gapped): `telemetryPipeline.cold.local.existingClaim`, or `cold.local.create.enabled`. Ingest writes it and the server reads it, so it must be ReadWriteMany when those pods run on more than one node.
- **S3**: `telemetryPipeline.cold.s3.bucket`, `region` and `kmsKeyId`. The pods' identity needs access to the bucket and key.

Secret keys it needs:

| Secret | Key | Used as |
|---|---|---|
| `secrets.ingest` | `NI_INGEST_SERVICE_TOKEN` | The bridge's token to ingest (read from the same Secret, so the two never differ) |
| `secrets.bridge` | `NI_BRIDGE_STREAM_SERVICE_TOKEN` | The bridge's token for the server's internal device lookup |

A missing key keeps the bridge pod from starting, rather than accepting events it cannot deliver.

### What changes in the release

- The bridge becomes a StatefulSet with one PersistentVolumeClaim per pod (`stream-wal-<release>-bridge-<n>`, mounted at `/var/lib/keystone/bridge/stream`). Claims are kept when a pod is deleted or scaled away, so events a pod accepted are delivered when a pod with the same ordinal comes back.
- The autoscaler only scales the bridge up. To scale down, lower `bridge.hpa.min`, scale the StatefulSet by hand once the pods above the new count show `keystone_bridge_stream_queue_pending_records` at 0, then delete their claims yourself.
- Ingest gains a headless Service the bridge resolves to split organisations over the replicas.
- New NetworkPolicies allow bridge to ingest (8081) and bridge to server (8080).

The claim template is immutable: to grow `bridge.stream.persistence.size` later, expand the existing claims with a StorageClass that allows it, and delete the StatefulSet with `--cascade=orphan` before the upgrade.

## Back-pressure, never a silent drop

- When the WAL volume's free space would fall below the floor (`bridge.stream.walMinFreeBytes`, 2 GiB in the chart), the log reaches its maximum (`walMaxBytes`, 40 GiB), or the disk is full, the intake answers `503` with `Retry-After`, nothing of the request is accepted, and readiness turns false so the load balancer sends producers to other pods.
- A per-organisation token bucket (`NI_BRIDGE_STREAM_ORG_RATE`, default 20,000 events per second, burst twice that) answers `429` so one account cannot crowd out the others.
- A record ingest refuses permanently is kept in a dead-letter log with its reason, counted on `/metrics`, and can be replayed.
- After a crash, only the last segment's unsynced tail can be torn; it is cut and kept under `quarantine/`. Damage anywhere else refuses to open the log rather than skip records already reported durable; the other bridge replicas keep taking traffic.
- On shutdown the bridge stops intake, delivers for up to 25 seconds (`bridge.stream.drainTimeout`) and leaves the rest on its volume for the next pod. Keep `bridge.stream.terminationGracePeriodSeconds` (60) above it.

## The intake API

`POST /api/v1/telemetry/events` on the bridge, body `{"events": [...]}`, at most 10,000 events and 10 MiB.

| Field | Rule |
|---|---|
| `id` | UUID, optional (one is assigned) |
| `timestamp` | RFC 3339, optional; within 14 days before and 2 hours after now |
| `source`, `type` | Required, at most 64 bytes |
| `device_id`, `user_id`, `session_id`, `trace_id` | At most 128 bytes |
| `severity` | At most 16 bytes |
| `project_id` | UUID |
| `org_id` | Service producers only |
| `payload` | A JSON object of at most 256 KiB |

Answers: `202 {"accepted": n, "rejected": [{"index", "reason"}]}`; `400` or `413` for a malformed or oversized request; `401 unauthenticated` or `device_inactive`; `429` or `503` with `Retry-After`, nothing accepted.

Agents point at the bridge with `KEYSTONE_TELEMETRY_URL`.

## Settings

| Variable | Default | Plane |
|---|---|---|
| `NI_TELEMETRY_PIPELINE_ENABLED` | `false` | ingest, server |
| `NI_BRIDGE_STREAM_ENABLED` | `false` | bridge |
| `NI_BRIDGE_STREAM_WAL_DIR` | `/var/lib/keystone/bridge/stream` | bridge |
| `NI_BRIDGE_STREAM_WAL_MIN_FREE_BYTES` | 512 MiB (chart: 2 GiB) | bridge |
| `NI_BRIDGE_STREAM_WAL_MAX_BYTES` | 0, only the free-space floor (chart: 40 GiB) | bridge |
| `NI_BRIDGE_STREAM_SERVICE_TOKEN` | none: service producers off | bridge |
| `NI_BRIDGE_STREAM_ORG_RATE` | 20,000 | bridge |
| `NI_BRIDGE_STREAM_INGEST_URLS` or `NI_BRIDGE_STREAM_INGEST_DNS` | one required | bridge |
| `NI_BRIDGE_STREAM_INGEST_TOKEN` | required | bridge |
| `NI_BRIDGE_STREAM_DRAIN_TIMEOUT` | `25s` | bridge |
| `NI_TELEMETRY_HOT_RETENTION` | `168h` | ingest |
| `NI_TELEMETRY_COLD_DIR` | `/var/lib/keystone/telemetry-cold` | ingest, server |
| `NI_TELEMETRY_COLD_S3_BUCKET`, `_REGION`, `_ENDPOINT`, `_KMS_KEY_ID` | local directory | ingest, server |
| `NI_TELEMETRY_COLD_RETENTION_DAYS` | 365 | ingest |
| `NI_TELEMETRY_COMPACTOR_DISABLED` | `false` | ingest |
| `NI_TELEMETRY_DATABASE_URL` | the main database | ingest, server |
| `NI_TELEMETRY_SHARDS` | none | ingest, server |

### A database of its own

By default telemetry lives in the main database. `NI_TELEMETRY_DATABASE_URL` (chart: `telemetryPipeline.database`) moves it to one dedicated PostgreSQL, and `NI_TELEMETRY_SHARDS` (`s1=postgres://...,s2=postgres://...`; chart: `telemetryPipeline.shards`) spreads organisations over several. Ingest applies the telemetry schema to each at start. Shard names are the placement: never rename one.

## Metrics

`keystone_bridge_stream_*` (intake, append latency, queue, delivery, dead letters, endpoint health), `keystone_ingest_stream_*` (records, duplicates, write and end-to-end latency), `keystone_telemetry_compactor_*`, `keystone_telemetry_cold_sweeper_*`.