Telemetry pipeline

The telemetry pipeline is Govern's own event stream and telemetry store for agent and service events. Its only stateful dependencies are PostgreSQL and object storage: an S3 or S3-compatible bucket, or a local directory for self-hosted and air-gapped installations. Kafka is not needed.

It is off by default. With it off, nothing changes.

How an event travels

1. A producer posts events to the bridge: POST /api/v1/telemetry/events, with a device token or a service token. The organisation comes from the credential.
2. The bridge (keystone-bridge) validates each event and appends it to a write-ahead log on its own volume. It answers 202 only after the write is on disk (fsync).
3. The bridge delivers the log to the ingest replicas (keystone-ingest), each organisation always to the same replica.
4. Ingest writes the events into hourly PostgreSQL partitions, dropping duplicates by event id. The bridge acknowledges only when every part of a batch is stored.
5. A compactor turns each closed hour into one sealed Parquet file per organisation (zstd, SHA-256 and HMAC-SHA256 under NI_AUTH_AUDIT_HMAC_KEY), reads it back and verifies it before it records it.
6. Compacted hours older than the hot retention (NI_TELEMETRY_HOT_RETENTION, default 168 hours) are dropped from PostgreSQL. Cold files are kept for the organisation's cold retention.

Delivery is at least once and duplicates are dropped at ingest, so no accepted event is lost or stored twice.

Reading telemetry

Events are searched in the Observe log group /ni/telemetry (FilterLogEvents, GetLogEvents, CreateExportTask), which reads both tiers and never reads an hour twice or skips one. Every string passes the account's DLP redaction. The console page is Observe > Telemetry, with status (Healthy, Delayed, Not receiving with the server's reason), the hot window, the archive per day and recent sources.

Turn it on with Helm

telemetryPipeline:
  enabled: true                  # also turns on bridge.stream
  hotRetention: 168h
  cold:
    local:
      create:
        enabled: true            # a ReadWriteMany claim <release>-telemetry-cold, kept on uninstall
        size: 500Gi
bridge:
  stream:
    persistence:
      size: 50Gi

Exactly one cold store is required; the chart refuses none or both:

• A directory (self-hosted, air-gapped): telemetryPipeline.cold.local.existingClaim, or cold.local.create.enabled. Ingest writes it and the server reads it, so it must be ReadWriteMany when those pods run on more than one node.
• S3: telemetryPipeline.cold.s3.bucket, region and kmsKeyId. The pods' identity needs access to the bucket and key.

Secret keys it needs:

A missing key keeps the bridge pod from starting, rather than accepting events it cannot deliver.

What changes in the release

• The bridge becomes a StatefulSet with one PersistentVolumeClaim per pod (stream-wal-<release>-bridge-<n>, mounted at /var/lib/keystone/bridge/stream). Claims are kept when a pod is deleted or scaled away, so events a pod accepted are delivered when a pod with the same ordinal comes back.
• The autoscaler only scales the bridge up. To scale down, lower bridge.hpa.min, scale the StatefulSet by hand once the pods above the new count show keystone_bridge_stream_queue_pending_records at 0, then delete their claims yourself.
• Ingest gains a headless Service the bridge resolves to split organisations over the replicas.
• New NetworkPolicies allow bridge to ingest (8081) and bridge to server (8080).

The claim template is immutable: to grow bridge.stream.persistence.size later, expand the existing claims with a StorageClass that allows it, and delete the StatefulSet with --cascade=orphan before the upgrade.

Back-pressure, never a silent drop

• When the WAL volume's free space would fall below the floor (bridge.stream.walMinFreeBytes, 2 GiB in the chart), the log reaches its maximum (walMaxBytes, 40 GiB), or the disk is full, the intake answers 503 with Retry-After, nothing of the request is accepted, and readiness turns false so the load balancer sends producers to other pods.
• A per-organisation token bucket (NI_BRIDGE_STREAM_ORG_RATE, default 20,000 events per second, burst twice that) answers 429 so one account cannot crowd out the others.
• A record ingest refuses permanently is kept in a dead-letter log with its reason, counted on /metrics, and can be replayed.
• After a crash, only the last segment's unsynced tail can be torn; it is cut and kept under quarantine/. Damage anywhere else refuses to open the log rather than skip records already reported durable; the other bridge replicas keep taking traffic.
• On shutdown the bridge stops intake, delivers for up to 25 seconds (bridge.stream.drainTimeout) and leaves the rest on its volume for the next pod. Keep bridge.stream.terminationGracePeriodSeconds (60) above it.

The intake API

POST /api/v1/telemetry/events on the bridge, body {"events": [...]}, at most 10,000 events and 10 MiB.

Answers: 202 {"accepted": n, "rejected": [{"index", "reason"}]}; 400 or 413 for a malformed or oversized request; 401 unauthenticated or device_inactive; 429 or 503 with Retry-After, nothing accepted.

Agents point at the bridge with KEYSTONE_TELEMETRY_URL.

Settings

A database of its own

By default telemetry lives in the main database. NI_TELEMETRY_DATABASE_URL (chart: telemetryPipeline.database) moves it to one dedicated PostgreSQL, and NI_TELEMETRY_SHARDS (s1=postgres://...,s2=postgres://...; chart: telemetryPipeline.shards) spreads organisations over several. Ingest applies the telemetry schema to each at start. Shard names are the placement: never rename one.

Metrics

keystone_bridge_stream_* (intake, append latency, queue, delivery, dead letters, endpoint health), keystone_ingest_stream_* (records, duplicates, write and end-to-end latency), keystone_telemetry_compactor_*, keystone_telemetry_cold_sweeper_*.