Evals

neuralinverse eval measures the agent on a set of small, realistic coding tasks, so that a change to the prompt, tools or loop behaviour can be compared against the current defaults on the same tasks, with confidence intervals.

Status. No real-model results have been recorded yet. The scripted solvers confirm that every task is solvable through the agent's tools and that doing nothing fails every task, but they say nothing about model quality.

neuralinverse eval list                                   # the tasks and variants
neuralinverse eval run --model anthropic/claude-sonnet-5-5 # run the tasks, write results and a report
neuralinverse eval report evals/results/<run>             # render a report again
neuralinverse eval validate                               # check that every task is a real test

The tasks live in the CLI's source (evals/tasks, 24 tasks), so run evals from a checkout of the CLI. From a checkout, npm run eval -- <args> runs the same command without building.

How a run works

For each task, model, variant and repetition:

1. The task's starting repository is copied to a temporary folder and committed as a fresh git repository; the task's setup steps run there.
2. A real agent runtime starts headlessly in that folder: full-auto permissions, the operating-system sandbox when available (--no-sandbox turns it off), a private configuration folder (your project settings and user instructions are not used; providers from ~/.neuralinverse/settings.json and keys from the environment are), and the task's cost and time limits.
3. The task prompt is sent once; the agent works until it finishes, stops or hits a limit.
4. Hidden files are copied over the result, so the agent never saw or edited them, and the checks run.
5. One JSON line is appended to results.jsonl: status (pass, fail, error, skipped), every check with its output, turns, tool calls, tool errors, harness events, tokens, cost, time, the stop reason and the final answer.

Runs already in results.jsonl are skipped, so running the same command with the same --out resumes an interrupted evaluation.

Options

neuralinverse eval run --model anthropic/claude-sonnet-5-5 --model openai/gpt-5 \
  --variant baseline --variant prompt-v2 --repeat 3 --concurrency 4 --out evals/results/prompt-v2

Reports

A report gives pass@1 per model and variant (the share of runs whose checks all passed) with a 95% Wilson interval, error and skip counts, median and total cost, median time, turns and tool calls; pass@1 per task category; a matrix of tasks by configuration; and, with --baseline, every task and configuration that regressed or improved.

Read the intervals before drawing conclusions: with 24 tasks and one run each, a 95% interval around 70% spans roughly 50% to 85%. Use --repeat, and compare variants on the same models.

Variants

A variant is a named settings overlay:

{
  "name": "prompt-v2",
  "description": "System prompt v2: workflow guidance and per-family notes.",
  "settings": { "prompt": { "version": "v2" } }
}

Offline runs

• scripted:reference solves each task from its reference solution through the real tools, exercising the runtime, tools, checks and reports without a network.
• scripted:noop gives up at once; every task must fail with it, which proves the checks cannot be passed by doing nothing.
• replay:<file> replays a transcript written with --record, with the tools running for real; a run that diverges from the recording ends with an error.

Add a task

evals/tasks/<id>/
  task.json          # the task definition
  repo/              # the starting repository
  hidden/            # files copied over the result before grading
  solution.patch     # reference fix as a git diff against repo/
  solution.answer.md # reference answer for questions

{
  "id": "pagination-off-by-one",
  "title": "Fix off-by-one errors in pagination",
  "prompt": "Users report that page 1 skips the first results ... Fix it.",
  "category": "bugfix",
  "difficulty": "easy",
  "language": "typescript",
  "requires": ["python3"],
  "timeoutMs": 600000,
  "maxCostUsd": 1,
  "setup": ["npm ci"],
  "checks": [
    { "kind": "command", "run": "node --test test/*.test.ts", "expectExit": 0 },
    { "kind": "file-contains", "path": "src/a.ts", "pattern": "ceil", "negate": false },
    { "kind": "answer-contains", "pattern": "token[ -]?bucket" }
  ],
  "hidden": ["test/hidden.test.ts"]
}

• category is one of bugfix, feature, refactor, explain, test, multi-file, config, debug. requires lists programs that must be on PATH, or the task is skipped.
• Check kinds: command (exit code), file-contains (regular expression; negate to require absence), answer-contains (regular expression against the final answer).
• Tasks must run without network access.

Then run neuralinverse eval validate --tasks <id>: it checks that the task's checks fail on the original repository and pass with the reference solution.