Inference Public

Software engineering

Who tests the tester? Separation and evidence in the AI software lifecycle

How public-sector teams can keep AI-built code and tests honest: service rules first, protected acceptance checks, evals and a release record.

Two engineers reviewing a release candidate at their desks

A resident submits an application to a council. The browser times out before the receipt appears. They try again. If the first submission was recorded, the service should return the existing receipt and leave one application on the record.

A test that checks for a receipt after an ordinary submission can pass while the retry creates a duplicate. A coding agent might write that test, run it and report that the feature works. An engineer could make the same mistake. With AI-assisted delivery, a team can produce code and tests quickly enough for the two to agree before anyone has checked what either says about the resident’s experience.

The council example is illustrative. It shows what we want a delivery team to be able to explain at Inference Public. Which service rule did you test? How do you know the test would catch a breach? Who decided the evidence was enough to release?

Start with the service rule

Two different things can go wrong when an agent builds a feature and checks its own work. It can read its implementation and mistake what the code does for what the service ought to do. It can also optimise for a passing check by weakening the check itself. A 2026 preprint by Konstantinou, Tambon and Papadakis found that models detected fewer faults when they had already produced the faulty code. In a separate set of agent evaluations, METR observed models modifying tests and scoring code to earn higher scores. Both findings come from experimental settings. They give delivery teams a reason to check how their own evidence is produced. Read the test generation study and METR’s account of reward hacking.

Start with the expected behaviour in terms the service owner can check. What counts as a recorded application? What should a resident see if the first attempt succeeded but the response was lost? What should happen if the service is still unsure? If the answer is unsettled, take it back to the people who own the service outcome. Whatever the current implementation happens to do is a poor way to settle a policy or operational question.

For changes where failure would have significant consequences, have someone derive the acceptance checks from those agreed rules before inspecting the candidate implementation. Keep ordinary developer tests visible so the build team can work quickly. Protect the acceptance suite, fixtures and runner from changes by the build agent. Reserve some cases for the release run so the gate also exercises behaviour the agent has had less opportunity to tune against. The gate executes the approved checks itself. If a check fails, give the build team enough information to reproduce and fix the problem. If a check needs changing, record who approved the change and why.

A different model is an option. In one Elixir study, switching models made no significant difference to the number of faults detected when the generated tests could see the implementation. The human-written suites found more faults in that study and were written without seeing the candidate code. The comparison varied both who wrote the test and what they could see. For a delivery team, the more immediate questions are where the expected behaviour came from and who can change the check. Read the Elixir study.

Show the test can catch the failure

Take the council retry scenario. Run the acceptance test against a deliberately faulty version that creates a second application after a timeout. The check should fail because it found two stored applications. Run the same check against the candidate build and verify that it records one application and returns its existing receipt. A test that goes red because a fixture is missing tells the reviewer something quite different.

There are several ways to do this in practice. Meta used AI to propose plausible faults and generate tests that exposed them. In trials at Messenger and WhatsApp, engineers accepted 73% of the tests reviewed for usefulness, while about 36% of the tests rated for relevance were considered possibly or definitely related to the privacy concern being investigated. An engineer can reasonably keep a useful regression test and still ask for better evidence on the specific risk. Google learned a related lesson from mutation testing. After its developers initially rated 85% of reported mutants as unproductive, the team used developer feedback to build heuristics that suppress unproductive mutants, bringing that figure down to 11%. Read the Meta deployment study and Google’s experience with mutation testing.

Teams have also tested whole journeys and checked implementations against small executable specifications. At Momentic, a separate QA agent receives the requirements, tries the running application and returns reproducible findings to the build team. In a human-reviewed audit of 1,000 AI-authored unit tests in its own codebase, Momentic found only 9.6% were useful verification checks. Duplication and heavy mocking were recurring problems. Amazon engineers working on S3’s ShardStore used executable reference models to express how its storage system should behave through events such as crashes. The team reported catching 16 issues before production, including defects in the reference models themselves. These are accounts of particular systems. For a council service, the question is which combination would expose the failures that matter to residents and staff. Read Momentic’s practitioner account and the ShardStore experience report.

Bring tests and evals into the release decision

Suppose the same council adds an AI feature that drafts application summaries for staff. The team can write an executable test for a rule such as requiring an authorised officer to approve a draft before it changes the case record. It also needs evidence about the drafts themselves. Do they preserve what the resident wrote? Do they leave uncertain facts uncertain? What happens when two documents disagree or a page is missing?

An evaluation, or eval, gives the feature a set of realistic cases and judges its responses against agreed criteria. Use executable checks where the expected result can be stated precisely. Where judgement is needed, ask service experts to assess examples and use those judgements to check any automated grader. Include plausible summaries that leave out a material detail or turn an allegation into an established fact. If the grader consistently misses those errors, its score deserves less weight in the release decision. Read GOV.UK guidance on testing generative AI and Anthropic’s guidance on combining graders.

Agree the consequences before running the checks. A failed mandatory rule stops the release. An eval can also stop it when it falls short of a requirement agreed in advance. Compare the candidate with the current version on the same cases, and agree in advance when to run cases more than once. Look at results for the kinds of case where a mistake would have the most serious consequences. A high average can hide repeated failures on cases with missing information. If required evidence is missing or a consequential finding is unresolved, the release decision stays open. A change to the model, prompt or source material can change the service’s behaviour, so rerun the relevant checks and record the configuration that was actually evaluated.

The principle applies when AI is used only to build the software, too. A reviewer agent can spot that a test checks for a receipt without checking that the application was stored. That finding is a lead. Reproduce it or have a reviewer confirm it before using it to decide whether the release can go ahead.

Give the approver a record they can question

For a consequential change, the release owner should be able to see:

If the visible checks pass and a held-back check fails, put the failed behaviour in front of the reviewer. It might reveal a missed interaction, an ambiguous rule or a problem in the test. SpecBench uses the visible-to-held-back gap as a benchmark measure and includes a case where the gap came from previously untested behaviour. In a release record, the failed expectation and its disposition matter more than the size of the gap. Read SpecBench.

The public body remains accountable for the service. A release record can feed a departmental service assessment or a council’s own assurance process, alongside evidence from users, accessibility testing and live operation. The cross-government quality engineering principles, introduced by GDS and the Legal Aid Agency and still in alpha, make the same point about proportionate effort. The more serious the consequence for users, the more the team should ask of the evidence. Read the cross-government quality engineering write-up.

What to ask a supplier

Ask a supplier to walk through one consequential rule from a recent change. Who agreed what should happen? Could the build agent change the check? Can the supplier show a faulty case that the check rejected? Who decided the candidate could go ahead and what did they have in front of them?

That conversation tells a commissioner more than a coverage figure. It gives them a way to see whether the supplier can connect what the service is meant to do with what the release evidence actually shows.


Research note. Konstantinou, Tambon and Papadakis report about 25% detection for independently generated tests and 14% when tests were generated after the model produced faulty code. They used Python benchmarks and, for each task, kept the faulty implementation that failed the fewest of the benchmark's reference tests, so the faults were not trivially exposed. The comparison describes what happened in that experiment. Read the study and its selection method.

All insights

Start a conversation

Let’s talk about what you need to build.

Bring the service, the constraint or the engineering question. We’ll work through a sensible starting point.

Discuss a project