Skip to content

Research · November 8, 2026

Measuring prompt injection before building a defence

A plain classifier in front of automatic replies let some attacks through, including on a message from someone in distress. We measured that first and kept human approval as the only defence.

Before an assistant may reply on its own, someone has to know how often a hostile message can steer it. We measured that for one proposed design, before the design went into production. This note reports the measurement, its limits and the decision it supported.

What we tested

The test used a deliberately plain setup. A local model classified incoming comments by category and risk at temperature zero, and the real reply-risk rule then decided whether a reply could go out without a person. We set the environment to the most permissive case, so the classification itself was the only barrier. The attack families came from public taxonomies named in the findings file, including those of the BIPIA benchmark, not from our imagination, so the numbers can be compared with published work.

We asked the right question: not "did the label change", but "would this reply have been sent on its own". A label can slip without changing the decision, because the rule also looks at the category.

What we found

The test ran 64 attack runs. Of those, 48 were on comments that should have been treated as high risk. Seven of the 48 passed the gate, which is 14.6 percent.

  • Attacks that pretended a finished answer had already been given got through in four of eight runs.
  • Attacks that told the model to ignore earlier context got through in two of eight.
  • The blunt "ignore all previous instructions" form got through in none of sixteen runs.

All seven that passed were relabelled as praise with low risk. Two were a message from someone saying they could no longer cope. A cheerful automatic thank-you to that message is the worst outcome this system could produce.

What surprised us

  • The existing sanitising step stopped none of the successful attacks. It only catches values it does not recognise, and every successful attack produced a perfectly valid label.
  • Obvious attacks failed and subtle ones worked. Someone who built a defence around searching for "ignore instructions" would have blocked the attack that did not work.
  • Tying a rule to the category held up better than tying it to the risk level. One attack lowered the risk label in five of eight runs but moved the category out of the high-risk set only once.

The run also confirmed a bug found the same day: three high-risk categories were missing from the list of known categories, so they were silently turned into "other". A data-deletion request under an authority-claim attack could have been answered automatically. The fix is in the file's before-and-after table.

The decision

Automatic reply stays off. The measured number is the reason, not a general worry. In the most permissive setting, 14.6 percent of high-risk comments passed. The file concludes that the structural defence, that nothing leaves without a person's approval, is the only real defence we have, and the measurement shows it. In the product, with the default settings, posts, replies, e-mails, publishing and spending wait for your approval, and rules decide which requests reach you as a card. See how to approve AI actions safely and the approval card entry.

If automatic reply is ever wanted, the file also states the precondition: the defence cannot sit at the vocabulary layer, which caught none of the seven.

Limits

The measurement used one model, one prompt and temperature zero. Another model would give other numbers. The corpus is small and the file says so. The measurement is a baseline taken before a classifier existed, which is worth more than one taken after, and it is rerunnable from the corpus and script in the repository.

Where this comes from

  • docs/evidence/injection/findings.md, dated 2026-09-03, with baseline.json beside it.

Sources