Skip to content

Research · November 8, 2026

What a nine billion parameter local agent passes, and what that score means

We ran a local nine billion parameter model through twelve task categories. One run kept a change out, and a later run raised the score. An audit of that second run says how much of the rise to believe.

When a model runs on your own computer, "does it work" has to be answered with a test, not a feeling. This note reports two of our internal runs of a local agent and, more usefully, what we concluded from them. These are our own tests on our own task suite. They say nothing about other products.

The setup

Both runs used the same suite: twelve task categories, such as web search, extraction, form filling, file operations, memory recall, Turkish casing and dates, and prompt injection. Each category ran six times, three in Turkish and three in English. The model was a nine billion parameter open model at four-bit quantisation, served by llama.cpp on one Apple Silicon Mac. The suite hash is recorded in the result files so a changed suite is visible.

Run one: a change held back

The first run compared the current agent core against a new loop design. The current core passed 40 of 72 runs and the new loop passed 38 of 72. We had written the promotion rule before the run: no category may drop by two or more, the total must be at least equal, and there must be no destructive action. The new loop failed the first two rules. The verdict in the file is HOLD.

The traces gave causes. In one category the new loop's planning step planned a purchase it should have declined, and a local judge rejected a correct refusal. In another, the model wrote Turkish city names with the wrong dotted and dotless letters. The file also records a destructive file deletion by the current core in three runs, which the new loop avoided. We kept the change out, and the file says so plainly.

One method lesson: at temperature zero the engine decodes greedily and ignores the seed, so earlier repeats had in effect been one run each. The run changed to the model card's sampling settings so the repeats could differ.

Run two: a higher score, read carefully

After fixes to the harness and the tools, the next run passed 66 of 72 on the same twelve categories, against 38 of 72 at the earlier baseline. The gate stayed closed: two categories were still four of six, below the 80 percent bar, and the target was a full pass plus a hidden set.

A separate audit of that run read all 72 traces. Its verdict is "partly". The 66 is a true score for this agent and this tool set on these fixtures, and it found no pass obtained without doing the work. But most of the rise came from harness and tool changes made after diagnosing those same tasks and seeds. The audit calls that a tuned-on-test score and says it is not evidence of generalisation. It also found that one category is not a model measurement at all, because a code check refuses before any model call, and that one click test landed at the same point six times.

What we conclude

  • A score on a suite you tuned against measures the suite. Keep a hidden set.
  • Write the promotion rule before the run, and let it say no.
  • Read the traces. The causes in run one were in the transcripts, not in the totals.
  • Six runs per category is few, and most intervals overlap. The run reports confidence intervals for that reason.

This is why we do not publish a single "local agent pass rate" on this site. For how approvals sit around such an agent, see the agents team entry and the validator entry.

Where this comes from

  • docs/evidence/lm/eval/m0-final-2.md, run one, dated 2026-10-31.
  • docs/evidence/lm/eval/c1a.md, run two, dated 2026-11-03.
  • docs/research/remote/c1a-benchmark-audit.md, the audit of run two, dated 2026-11-03.

Sources