---
title: "AI Gateway"
---

# AI Gateway

How the Neural Inverse Govern model gateway decides every model request, its OpenAI- and Anthropic-compatible endpoints, request and response headers, and refusal codes.

The model gateway is where Govern enforces model access. Every model request from the Neural Inverse apps, your applications, agents and routed coding tools goes through it, and it decides each one against your organisation's policy before anything reaches a provider. It runs inside the Govern server, next to the database, so budgets and rate limits hold across any number of replicas and provider keys never leave the process that holds the vault.

## Pages in this part

| Task | Page |
|---|---|
| Connect a provider, approve models and assign them | [Model access](https://neuralinverse.com/docs/govern/model-access/) |
| Gateway keys, keyless workload identity, coding tool tokens, authorizers | [Gateway keys and workload identity](https://neuralinverse.com/docs/govern/gateway-keys/) |
| Spend and token budgets, rate limits | [Budgets and rate limits](https://neuralinverse.com/docs/govern/budgets-rate-limits/) |
| Routes, fallbacks, retries, circuit breaker and response cache | [Routing and caching](https://neuralinverse.com/docs/govern/routing-caching/) |
| DLP and guardrails on prompts and answers | [Guardrails and DLP](https://neuralinverse.com/docs/govern/guardrails/) |
| Request logs, provider health and attribution | [Request logs and observability](https://neuralinverse.com/docs/govern/request-logs/) |

## Where to configure it

The console has two services for this:

- **Models**: Providers and keys, Model catalog, Routing, Budgets, Assistant, and a Gateway group with Gateway endpoints, Connect a coding tool, Routes and strategies, Response cache, Guardrails, Rate limits, Request log settings and Provider health. Request logs, LLM traces and Model usage are under Monitor.
- **AI Gateway**: Gateways, Endpoints and domains, Gateway keys, Workload identity, Authorizers, Protection, Usage plans, Policies, Routing and reliability, Regions, Self-hosted nodes, governance pages and observability pages.

The in-server gateway described in these docs is working. The AI Gateway service also manages dedicated gateways with their own host names served by a separate data plane; the deployment of that data plane is recorded as partial in the source, and self-hosted gateway nodes are partial too. These docs describe the in-server gateway.

## How a request is decided

Every request to `/api/v1/models/gateway/v1/*` goes through the same steps, in order:

1. **Authenticate** the caller (`models:read`) and parse the body.
2. **Resolve** the caller's teams, project, classification, policy, catalog, assignments, settings and routes in one read-only transaction.
3. **Find the candidates**: the named model and its fallback chain, or the targets of the named route. Each candidate must be usable by the caller for the request's use and classification.
4. **Apply DLP** to the prompt at the level the classification requires: redact or refuse.
5. **Check rate limits**, then the response cache.
6. **Dispatch** to each candidate in turn: skip it while its circuit is open, check its rate limits, reserve the worst case on every budget, call the provider, retry transient failures.
7. **Relay** the answer in the caller's wire format, settle actual usage, audit the attempt, store the answer in the cache when allowed.

Steps 1 to 4 fail closed. Caching, retries and routing only save or repeat a provider call; none of them can let through a request that steps 1 to 4 refuse.

## Endpoints

The base URL is your API URL followed by `/api/v1/models/gateway`. Anthropic clients use the base URL as it is; OpenAI clients add `/v1`.

| Method and path | Format |
|---|---|
| `POST /api/v1/models/gateway/v1/chat/completions` | OpenAI chat completions |
| `POST /api/v1/models/gateway/v1/completions` | OpenAI legacy completions |
| `POST /api/v1/models/gateway/v1/embeddings` | OpenAI embeddings |
| `POST /api/v1/models/gateway/v1/responses` | OpenAI Responses API |
| `GET /api/v1/models/gateway/v1/models` | Models and routes the caller may use |
| `POST /api/v1/models/gateway/v1/messages` | Anthropic Messages |
| `POST /api/v1/models/gateway/v1/messages/count_tokens` | Anthropic token counting |
| `POST /api/v1/models/gateway/v1/prompts/{prompt}/chat/completions` | Prompt library template, OpenAI format |
| `POST /api/v1/models/gateway/v1/prompts/{prompt}/messages` | Prompt library template, Anthropic format |
| `POST /api/v1/models/gateway/v1/rerank` | Rerank (`openai_compatible` and `ollama` connections) |
| `POST /api/v1/models/gateway/v1/images/generations` | OpenAI image generation |
| `POST /api/v1/models/gateway/v1/audio/transcriptions` | OpenAI transcription |
| `POST /api/v1/models/gateway/v1/audio/speech` | OpenAI text to speech |

`GET /api/v1/models/gateway/endpoints` lists the same table as your deployment serves it, with full URLs. Request bodies are limited to 32 MiB. The Responses API accepts only `function` tools: tools the provider runs itself (web search, file search, code execution, remote MCP servers, computer use) are refused, because their traffic would never pass through DLP and the guardrails.

Provider-native paths are available for clients pinned to a provider's own API: `POST /api/v1/models/gateway/anthropic/v1/messages` and `POST /api/v1/models/gateway/openai/v1/chat/completions` (also `/v1/completions`, `/v1/embeddings`, `/v1/responses`). They run through the same checks and are off until an administrator turns them on; see [Gateway keys and workload identity](https://neuralinverse.com/docs/govern/gateway-keys/#provider-native-passthrough).

### Example

```bash
curl -sS "$KEYSTONE_URL/api/v1/models/gateway/v1/chat/completions" \
  -H "Authorization: Bearer $GATEWAY_KEY" \
  -H "X-Keystone-Project: $PROJECT_ID" \
  -H "X-Keystone-Use: chat" \
  -H 'Content-Type: application/json' \
  -d '{"model": "claude-sonnet-5-5", "messages": [{"role": "user", "content": "Summarise this change."}]}'
```

The `model` field is a model key from your catalog or a route key.

## Who can call it

The gateway accepts every Govern credential: a signed-in person's token, OAuth access tokens, API keys and agents, as long as the caller holds `models:read`. It adds its own:

| Credential | For |
|---|---|
| Gateway keys | Applications. A key that can call the gateway and nothing else, with its own scope, models, budgets and limits. |
| Workload identity | Workloads that exchange a token they already have (Kubernetes, CI, a Govern agent, a client certificate) for a short-lived gateway key. |
| Gateway tokens | Coding tools used by a person: a short-lived token bound to the person's session or enrolled device. |
| Authorizers | Per-route decisions by JWT, a custom HTTP or CEL authorizer, or none, for traffic without a Govern credential. |

See [Gateway keys and workload identity](https://neuralinverse.com/docs/govern/gateway-keys/).

## Request headers

| Header | Effect |
|---|---|
| `X-Keystone-Project` | Decide the request in this project: its classification, policies, assignments and budgets |
| `X-Keystone-Use` | `chat`, `agent`, `completion` or `embedding`; the rerank, image and audio endpoints use `rerank`, `image` and `audio` |
| `X-Keystone-Classification` | Raise the request's classification; it can never lower it |
| `X-Keystone-Metadata` | A JSON object of attribution, at most 16 keys and 8192 bytes |
| `X-Keystone-Tags` | Up to 10 tags |
| `X-Keystone-Retries` | Lower the retries of this request |
| `X-Keystone-Cache-Control` | `no-cache`, `no-store` |
| `X-Keystone-Client` | The calling app and version, for the audit trail (advisory) |
| `X-Keystone-Context-Request-Id` | The context retrieval this request used, linked in the evidence chain |

Metadata and tags are DLP-scanned like the prompt and never sent to the provider.

## Response headers

| Header | Value |
|---|---|
| `X-Keystone-Request-Id` | The request id (later attempts are `<id>.<n>`) |
| `X-Keystone-Model` | The catalog model that answered |
| `X-Keystone-Route` | The route, when the request named one |
| `X-Keystone-Attempts` | Upstream attempts made (0 for a cache hit) |
| `X-Keystone-Cache` | `hit`, `semantic_hit`, `miss`, `bypass` or `off` |
| `X-Keystone-Budget-Warning` | A budget passed its warning threshold or a soft limit |
| `X-RateLimit-Limit`, `X-RateLimit-Remaining`, `X-RateLimit-Reset` | When a rate limit rule applied |
| `Retry-After` | With `circuit_open` and `rate_limited` |
| `Server-Timing` | `gw;dur=<ms>`: the gateway's own time before the response headers |

## Refusal codes

Refusals use each wire format's own error shape and carry `X-Keystone-Request-Id`.

| Code | Meaning |
|---|---|
| `model_not_found`, `model_not_allowed`, `use_not_supported` | The model or route is unknown, not assigned to the caller, or not for this use |
| `classification_not_allowed`, `region_not_allowed` | The model may not serve this classification or region |
| `project_forbidden`, `project_required` | The caller is not in the project, or the organisation requires one |
| `dlp_blocked`, `dlp_unavailable` | DLP refused the prompt or answer, or could not run |
| `guardrail_blocked` | A guardrail refused the prompt or answer |
| `budget_exceeded`, `plan_limit` | A hard budget or the plan's limit would be crossed |
| `rate_limited` | A rate limit rule refused the request |
| `circuit_open` | Every candidate's circuit is open |
| `no_eligible_model`, `provider_disabled`, `provider_unavailable`, `provider_auth_failed`, `key_unavailable` | No usable model or provider |
| `gateway_suspended` | The organisation's gateway was suspended by the AI security kill switch |
| `invalid_request`, `service_unavailable` | Malformed request, or a dependency could not be reached |

A provider's own error body never reaches the caller, because it can quote the prompt. The caller gets a generic refusal and the request id; operators get the body with DLP redaction applied.

## For the Neural Inverse apps

The Editor and the Agents app fetch a signed view of what the person may use from `GET /api/v1/models/resolved?project=<project id>` and verify it with the public keys at `/.well-known/neuralinverse-models-keys.json`. The manifest only shapes what the app offers; the gateway decides every request again. The manifest signing key is set with `NI_AUTH_MODELS_SIGNING_KEY_FILE` (or `_PEM`) and `NI_AUTH_MODELS_SIGNING_KEY_ID`; without it the endpoint answers `503 models.signing_unavailable` and no unsigned manifest is served.

## Audit

Every attempt is audited as `models.gateway.request` with the decision, model, classification, tokens, cost, route and attribution, but no prompt content. Refusals are audited with their code. Configuration changes are audited in their own transaction.