---
title: "Spaces and sources"
---

# Spaces and sources

How Context spaces follow your organisation, how to connect GitHub repositories and issues through a GitHub App, how indexing works, and how people add personal documents.

## Spaces

A space holds the sources of one node of your organisation. Govern keeps one space for every live organisation, department, team, group and project, through a reconciliation job that runs every 10 minutes and once at start. Space names follow renames.

- When a node is deleted, its space is **archived**, never deleted: its sources and chunks are kept, but it is no longer served anywhere (lists, retrieval, lenses, snapshots of live content). A space whose node comes back is restored. Administrators list archived spaces with `GET /api/v1/context/spaces?archived=true`.
- A project registered moments ago gets its spaces at the next reconciliation, or at once with `POST /api/v1/context/project/space` and `{"projectId": "..."}` (needs `context:space:write` on the project space; idempotent).
- **Personal spaces** are created when a member first promotes a document (below). Only the owner can use them, in every rollout mode and for super admins too.

## Sources

| Kind | What it indexes | Credential |
|---|---|---|
| `codebase` | A GitHub repository at a branch (the default branch when none is given), with include and exclude paths | The organisation's GitHub App installation |
| `issues` | The repository's issues and pull requests (most recently updated first, at most 1,000), optionally its 50 most recent failed GitHub Actions runs | The same GitHub App, with extra permissions (below) |
| personal document | Markdown, plain text or a supported code file a member uploads to their personal space, at most 1 MiB of UTF-8 | None |

Other kinds (`docs` or wiki, conversations, regulatory, custom, Linear issues) are refused at creation: no connector exists for them.

### Create a source

```http
POST /api/v1/context/spaces/{spaceId}/sources
Content-Type: application/json

{
  "kind": "codebase",
  "name": "payments-api",
  "config": {
    "repoUrl": "https://github.com/acme/payments-api",
    "branch": "main",
    "includePaths": ["src/**"],
    "excludePaths": ["src/generated/**"]
  }
}
```

The request needs `context:space:write`. It answers `201` and queues indexing in the same transaction. An `issues` source takes `{"repoUrl", "installationId"?, "includeCiFailures"?}`. A source may also carry `classification` and `dlpAction`, which can only tighten what the space inherits ([Classification](https://neuralinverse.com/docs/context/classification/)).

`installationId` is optional. When it is omitted, Govern asks GitHub which installation of the organisation's App covers the repository. Errors: `422 context.github_app.not_configured` (no App), `422 context.source.installation_not_found` (the App is not installed on the repository), `503` (GitHub unreachable).

### Routes

| Method | Path | Permission |
|---|---|---|
| `GET` | `/api/v1/context/spaces` (`?nodeId=&nodeKind=`, `?archived=true`) | read |
| `GET` | `/api/v1/context/spaces/{spaceId}` | read |
| `GET`, `POST` | `/api/v1/context/spaces/{spaceId}/sources` | read, write |
| `DELETE` | `/api/v1/context/spaces/{spaceId}/sources/{sourceId}` | write |
| `POST` | `/api/v1/context/spaces/{spaceId}/sources/{sourceId}/reindex` | write (`202`) |
| `GET` | `/api/v1/context/sources/{sourceId}` | read |
| `GET` | `/api/v1/context/sources/{sourceId}/status` | read |
| `GET` | `/api/v1/context/index-jobs` (`?state=queued\|running\|retrying\|dead`) | admin |

## Connect GitHub

Govern reads private repositories only through a GitHub App that your organisation owns. Personal access tokens are not accepted.

1. **Create a GitHub App** with repository permissions **Contents: read** and **Metadata: read**, and the **push** webhook event. Nothing else and no write permission. For `issues` sources also grant **Issues: read** and **Pull requests: read**, and **Actions: read** for CI failures; without them those sources fail with GitHub's permission error.
2. **Mount a vault engine.** The App's key is kept in the organisation's Keystone Vault, in a KV version 2 engine mounted at `secret/` (the path is `secret/data/keystone/context/github-app` unless the operator changed `KEYSTONE_CONTEXT_GITHUB_APP_SECRET_PATH`). Without it, saving the App answers `409 context.github_app.vault_not_ready`.
3. **Save the App** (`context:space:admin` on the organisation's root space):

   ```http
   PUT /api/v1/context/github-app
   Content-Type: application/json

   {"appId": 12345, "privateKey": "-----BEGIN RSA PRIVATE KEY-----...", "webhookSecret": "..."}
   ```

   The key must be an RSA key of at least 2048 bits. Responses never contain it: they carry `appId`, `keyFingerprint` (as GitHub displays it), `enabled`, `version` and `webhookConfigured`. Saving again rotates the key; a rotation without `webhookSecret` keeps the stored secret. `GET` and `DELETE` on the same path read and remove the configuration.
4. **Set the webhook URL** of the App to `https://<your Govern host>/api/v1/context/github/webhook/<orgId>` with the same secret.
5. **Install the App** on the repositories you want indexed.

Each fetch reads the key from the vault, signs a short-lived App token and mints an installation token restricted to the one repository and to read access. Tokens are cached in memory until 5 minutes before expiry and never written to the database, logs or errors.

### Push webhook

`POST /api/v1/context/github/webhook/{orgId}` carries no Govern credential. It is authenticated only by `X-Hub-Signature-256` (HMAC-SHA256 with the organisation's webhook secret), checked before any other work, and rate limited per source address and per organisation.

- A push to a branch re-indexes every codebase source of the organisation with the same installation, repository and branch. Tags, deleted branches and other events are ignored.
- A delivery id already accepted within 72 hours answers `202 {"duplicate": true}` and re-indexes nothing.
- An unknown organisation, a missing secret and a bad signature all answer the same `401`.
- `GET /api/v1/context/github-app/deliveries` lists the 50 newest accepted push deliveries (kept 72 hours).
- Missed deliveries are caught by the maintenance scan.

Only hosts in `KEYSTONE_CONTEXT_GITHUB_HOSTS` (default `github.com`) are accepted, at creation and before every fetch, because the connector sends the GitHub token to that host. Add your GitHub Enterprise host there.

## Indexing

Sources are indexed by a background job. Retrieval keeps serving the previous version until a new one commits.

- **Versioned commits.** A run records the source's version before it fetches and commits only if no newer version was written meanwhile. A run that loses that race is re-queued, not failed. A shutdown during a run leaves the source as it was.
- **Failures.** Permanent failures (no credentials, `401`, `403`, `404`, invalid configuration, too large, nothing indexable) set the source to `error` without retry. Transient failures retry with backoff, then go to the dead-letter list; a dead job is run again with `POST .../reindex`. Any successful run clears `error`.
- **Idempotent.** When the content and the embedding settings equal the current version, nothing is re-embedded and no version is written.
- **Maintenance scan** every 15 minutes: sources indexed with another embedding model are marked stale and re-indexed; sources stuck in `pending` or `indexing` for 15 minutes with no job are re-queued.

### Chunking

| Content | How it is split |
|---|---|
| Markdown | By heading structure |
| Go | By declarations (functions, methods, types with their comments) |
| Every other detected language | Line windows of at most `KEYSTONE_CONTEXT_CHUNK_MAX_TOKENS` (default 400) tokens with 1/8 overlap |

A file that cannot be parsed falls back to line windows. Lines longer than the chunk size are split. Files with a NUL byte or invalid UTF-8 are binary and skipped; symbolic links and path-traversal entries are skipped.

Codebase sources skip by default `.git/`, `vendor/`, `node_modules/`, `dist/`, `build/` and `testdata/`, minified bundles, source maps, `*.pb.go`, `*_generated.go` and lockfiles. `KEYSTONE_CONTEXT_DEFAULT_EXCLUDES` replaces that list (`none` turns it off); a source's `excludePaths` apply on top.

Environment, secret, key and credential files are never fetched: the reader skips them by path before any content is read, so they are never chunked, embedded or stored. The rule list is the one described in [Agent instruction files](https://neuralinverse.com/docs/context/agent-files/).

### Repository configuration file

A `.keystone.yaml` or `.keystone.yml` file in any directory is always read and never indexed. The strictest `classification:` it declares raises the source's classification. An unknown value fails the run, and the previous version keeps serving.

### Code facts

For codebase sources the index job also records, per chunk: owners from CODEOWNERS; for Go, the qualified symbol, its direct callers, callees and tests; a summary of each `go.mod`; and the last commit that changed each file (best effort). Retrieval uses the call graph to append up to six related chunks after the top results (see [Retrieval](https://neuralinverse.com/docs/context/retrieval/)).

### Run report

`GET /api/v1/context/sources/{sourceId}/status` returns the queued or running job (`job.state`, `attempts`, `nextRunAt`) and `lastRun`: outcome (`indexed`, `unchanged`, `failed`), index version, revision, duration, files seen, indexed and skipped, up to 50 skipped files with their reason, chunks, chunks reused, whether chunks were embedded and why not (`dlp_action`, `external_endpoint`, `policy_unresolved`).

## Personal documents

A member adds a document to their own personal space:

```http
POST /api/v1/context/me/sources
Content-Type: application/json

{"name": "my notes", "path": "notes.md", "content": "..."}
```

- At most 1 MiB of UTF-8. Promoting the same name again replaces the content.
- Personal sources are private: only the owner retrieves them, they are never inherited by other spaces, they cannot be added to lenses, and snapshots skip them.
- Environment, secret and key files are refused with `400`.
- `GET /api/v1/context/me/sources` returns the personal space id and its sources.
- On the hosted service the number of personal sources is a counted quota (default 100). A self-hosted installation does not limit them.

## From the command line

The Keystone command-line tool shipped with the release calls the same API with `KEYSTONE_URL` and `KEYSTONE_TOKEN`; the server enforces every permission.

```bash
# Register the current repository as a project, its spaces and a codebase source
ni keystone init --team=platform --write-config

# Add a document to your personal context
ni keystone context promote --source=notes.md

# Register every repository of a GitHub organisation (GITHUB_TOKEN or GH_TOKEN)
ni keystone admin projects import --github-org=acme --team-map=teams.txt --dry-run
```

The project is detected from the nearest `.keystone.yaml` `project_id`, then the git `origin` remote. `--write-config` adds `project_id` to `.keystone.yaml` and never overwrites a different one. In the import team map, each line is `<repository or glob>: <team>` and the first match wins; archived, disabled and forked repositories are skipped by default.