--- title: "Retrieval and attribution" --- # Retrieval and attribution Retrieve context through the API, how results are ranked and budgeted, how embeddings are kept behind DLP, and how every retrieval and the model call that used it are recorded as evidence. ## Retrieve ```http POST /api/v1/context/retrieve Authorization: Bearer X-Keystone-Client: neuralinverse-editor/1.4.2 Content-Type: application/json { "query": "how are refunds authorised", "spaceId": "", "tokenBudget": 4096, "maxPerLevel": 5, "lensId": "", "sessionId": "", "excludeRelated": false, "sourceKinds": ["codebase", "issues"] } ``` The answer carries `chunks`, `sourcesSummary`, `tokensUsed`, `attributionId` and `requestId` (the key under which the attribution record is stored). The caller needs `context:space:search` on each space that contributes; organisation and user always come from the token, and a body naming another is refused. | Field | Rule | |---|---| | `tokenBudget` | Default 4096; values above 32000 are lowered to 32000; negative values are `422` | | `maxPerLevel` | Default 5; values above 50 are lowered to 50 | | `spaceId` | Optional. Without it, the organisation-level chain is used. A value that is not a UUID is `404 context.space.not_found` | | `sourceKinds` | Optional: `codebase`, `docs`, `conversations`, `issues`, `workspace_live`, `regulatory`, `custom`; an unknown kind is `422`. Restricts retrieval to those sources | | `excludeRelated` | `true` turns off the related-code chunks (below) | `GET /api/v1/context/capabilities` tells a first-party app, without administrator rights, whether retrieval is served and its maximum and default budgets. ## Ranking Search runs only over the authorised spaces, capped per level, and only over each source's current index version. - **Hybrid score.** `score = (0.7 × vector + 0.3 × text) × weight`, where `vector` is 1 minus the cosine distance and `text` a normalised full-text rank. A chunk found by both searches never ranks below one found by either alone. Without an embeddings endpoint, search is full-text only. - **Level weights.** The weight of the space level the chunk came from: project 1.20, team 1.00, group 1.00, department 0.92, organisation 0.85, personal 0.85 (subproject 1.40 and business unit 0.88 are defined but unused). An operator changes the defaults with `KEYSTONE_CONTEXT_LEVEL_WEIGHTS` (`level=weight,...`). An organisation overrides any level with `GET`/`PUT /api/v1/context/weights` (`{"overrides": {"project": 1.3}}`; `context:space:admin`; weights in (0, 10] with at most 4 decimals; `{}` resets). - **Related code.** For the top 3 ranked code chunks, up to 2 callers, callees and tests each (at most 6 in total) are appended after the ranked chunks with score 0 and `metadata.relatedTo` and `relatedRelation` (`calls`, `called_by`, `tested_by`). They come from the same authorised source, pass DLP and only use budget left by the ranked chunks. Call graphs exist for Go only. - **Cache.** The search step is cached for 60 seconds per organisation, authorised space set and query. Authorisation, DLP and the budget run on every request; a cache hit of a source that was deleted or re-indexed is discarded. ## Embeddings Vector search needs an OpenAI-compatible embeddings endpoint (`KEYSTONE_CONTEXT_EMBEDDINGS_URL`, `_MODEL`, `_DIMENSIONS`; see [Settings](https://neuralinverse.com/docs/context/settings/)). The default search backend computes exact cosine distance in SQL. The `pgvector` backend needs the extension installed in your PostgreSQL image and created as a superuser; with `KEYSTONE_CONTEXT_SEARCH_BACKEND=pgvector` and no extension, retrieval fails closed with `503`. Text sent to an embeddings endpoint can leave your control, so the classification model applies before anything is sent: - Sources whose DLP action is `drop` or `block_source` (confidential, restricted, regulated projects) are never sent to any embeddings endpoint; they are searchable through full-text only. - Confidential and restricted sources are never sent to an **external** endpoint. An endpoint is external unless its host is loopback or listed in `KEYSTONE_CONTEXT_EMBEDDINGS_INTERNAL_HOSTS`, or `KEYSTONE_CONTEXT_EMBEDDINGS_EXTERNAL=false` says otherwise. - Every other chunk, and the query text, passes the organisation's DLP redaction first; only the redacted text is embedded. - A DLP scan failure stops the run and sends nothing. `GET /api/v1/context/status` (organisation administrators) reports the search backend, whether embeddings are configured, the model, its dimensions and the endpoint's locality (`in_deployment`, `external`, `none`). It never returns URLs, keys or error text. ## Attribution Every successful retrieval is recorded in the observability module's HMAC-chained evidence chain, as one record of category `context_attribution`: - the request id, the user, the space, the lens and the source ids; - per chunk: chunk id, source id, the SHA-256 of the content delivered and the source's index version; - the client: `clientId` (the OAuth client of the caller's token, verified), and `clientName` and `clientVersion` from the `X-Keystone-Client: name/version` header (declared by the app, recorded but never trusted; a malformed header is dropped); - for team memory chunks, the entry and its accepted revision. The record is written asynchronously after the retrieval. If the event cannot be queued, the retrieval answers `503 context.attribution.failed`. | Method | Path | Returns | |---|---|---| | `GET` | `/api/v1/context/attributions/{requestId}` | The records of a retrieval request | | `GET` | `/api/v1/context/attribution/{attributionId}` | One record by the `attributionId` a retrieval returned | | `POST` | `/api/v1/context/attributions/{attributionId}/links` | Records that an AI response used the retrieval: `{"responseId": "..."}` (at most 256 characters); `204`, idempotent | Each is visible to the person who retrieved, or to a `context:space:admin` who can read the space; anyone else gets `404 context.attribution.not_found`. Because the record is written asynchronously, a call right after a retrieval can answer `404`: retry for a few seconds. The evidence chain is verified daily and on demand with `GET /api/v1/observability/chain/verify`, and listed with `GET /api/v1/observability/compliance-evidence` (see [Evidence](https://neuralinverse.com/docs/evidence/)). ## Joining a model call to its retrieval When the Editor, the Agents app or the CLI sends a model request through the Govern model gateway, it names the retrieval it used: | Request header | Rule | |---|---| | `X-Keystone-Context-Request-Id` | The `requestId` (or `attributionId`) of the retrieval: 1 to 128 characters of `A-Z a-z 0-9 . _ : -`, starting with a letter or digit. Anything else is `400 invalid_request` before anything is sent | | `X-Keystone-Metadata` | The same id may be sent as the `context_request_id` key; two different values are `400` | | `X-Keystone-Client` | `name/version` of the app, recorded only | None of these reaches the model provider. When the call is answered, the gateway appends a `context_attribution_link` record to the same chain, with the gateway request id as the response id, and returns the outcome in `X-Keystone-Context-Link`: | Value | Meaning | |---|---| | `linked` | The link record was appended | | `not_found` | No record of the caller's organisation has that id, after waiting up to `KEYSTONE_MODELS_CONTEXT_LINK_WAIT_MS` (default 750 ms) | | `denied` | The retrieval is another person's and the caller is not an auditor (`audit:read`), or the caller is not a person | | `error` | The records could not be read or written | A failed link never fails or changes the model call; it is audited as `models.gateway.context_link`. `GET /api/v1/models/logs?context_request_id=` lists the model calls that used a retrieval.