An assistant writes a function, then writes six tests. All six pass. The engineer now has two pieces of code agreeing with each other, which is encouraging. Whether either agrees with the requirement remains a separate question.
This is not a reason to avoid assisted engineering. It is a reason to keep an independent account of the behaviour we intended. A test that repeats the implementation's misunderstanding can be thorough, readable and completely unhelpful.
Write the decision before the function
Consider a deliberately small example: a release may proceed only when every required check is recorded as passed and the responsible person has approved it. A missing check must block the release. A running check must block it too. The list of required checks must not be empty by accident.
That requirement contains a distinction a plausible implementation can miss. 'No recorded check has failed' does not mean 'every required check has passed'. An empty results object contains no failure. It contains no evidence of success either.
Before looking at the proposed code, write the expected decisions:
| Required results | Approval | Expected result |
|---|---|---|
| Unit passed; integration passed | Present | Allow |
| Unit passed; integration missing | Present | Block |
| Unit passed; integration running | Present | Block |
| Unit passed; integration failed | Present | Block |
| Both passed | Absent | Block |
| Required-check configuration empty | Present | Reject configuration |
These cases come from the requirement, not from branches discovered in the generated function. The included executable example uses this table. It is a teaching example, not a production deployment-control implementation.
Show that the test can disagree
Run the tests against an intentionally weakened implementation that allows a release whenever approval exists and no recorded status equals FAILED. The missing-check and running-check cases should expose it. If the suite stays green, find out why before adding more tests.
This is the practical value of a mutation exercise: make a realistic error and ask whether the tests notice. It does not establish that the suite detects every relevant defect. Choose mutations that reflect the risks of the actual change: removing a boundary check, reversing an authorisation condition, dropping a durable write or acknowledging a message too early.
For a bug fix, there is another useful demonstration. Run the regression case against the pre-fix revision and show the relevant failure, then run it against the fix. A test that passes both versions may still be useful for other behaviour, but it has not demonstrated this defect.
Inspect the fixtures and the oracle
Test data can quietly make the difficult case impossible. Every mock user has permission. Every identifier is unique. Every dependency answers immediately. The fixtures have arranged a comfortable world in which the code is correct.
Ask where the expected answer comes from. A hand-reviewed decision table, a contract agreed with another system and a known historical incident can each provide an independent basis. Calculating the expected value by calling the same helper as the implementation is often weaker than it looks.
Use parametrised cases to make boundary differences visible. Property-based testing can explore wider input spaces once you can state a property precisely. Neither method supplies the property or the correct expected behaviour for you. Those still require engineering judgement.
Test beyond the mock boundary
A mocked database may accept two concurrent writes that the actual schema rejects, or reject a situation that a real transaction can handle. A mocked queue cannot establish how your worker behaves when processing succeeds and the acknowledgement is lost.
Choose the layer that can reveal the failure. A small unit test is appropriate for a pure decision. A concurrency guarantee needs tests against the relevant transaction and constraint behaviour. An integration boundary needs contract tests and representative failure responses. A release path needs an actual deployment observation.
Keep clocks and randomness controlled where the test requires repeatability, but also exercise delayed, duplicated and out-of-order work when those conditions are part of the service contract. A retry test that only repeats an immediate successful call leaves most of the risk untouched.
Review what coverage cannot say
Line coverage can show that code was executed. It does not establish that the assertion would detect the wrong result. A test that calls a function and asserts that its response exists can cover every line while proving very little.
For a significant change, retain a short evidence note: the behaviour being changed, cases derived independently, negative cases, the realistic defect the suite detects, integration evidence and remaining uncertainty. Tie it to the revision that was reviewed.
Do not turn a mutation score or a count of generated tests into a productivity target. A team can improve the number while losing sight of the service. DORA's delivery measures are useful for understanding the broader delivery system; they do not validate an individual test suite.
Keep a person accountable for the claim
The engineer approving the change should be able to explain what the tests establish and where their boundary ends. AI assistance can help propose cases, generate data and inspect a failure. The acceptance decision still needs a defensible basis outside the assistant's confidence.
Start with the highest-consequence behaviour in one current change. Write its decision table, introduce one plausible defect and watch the test fail for the right reason. That is a much better first question than how quickly another hundred tests can be generated.
See pipeline evidence and our software engineering capability, or discuss a service or integration.
