Your AI Agent Passed Every Test. Who Verified the Tests?
Why passing tests is not the same as establishing correctness in AI-assisted software engineering.
An AI coding agent implements a feature, generates unit tests, executes them, and reports:
"All tests passed. Implementation complete."
The build is successful. Test coverage looks reasonable. The CI pipeline is green.
But here's a question worth asking:
What if the implementation and its tests are based on the same incorrect assumption?
Tests verify the conditions and expectations encoded within them. They cannot automatically detect requirements that were never considered. As AI agents increasingly handle implementation and test generation together, this creates an important architectural challenge:
How do we establish that the evidence used to validate an implementation is trustworthy?
1. The Correlated Assumption Problem
Consider a distributed job scheduler.
An AI agent receives a requirement: "Ensure that only one worker executes a job at a time."
The agent implements lease-based ownership and generates tests for lease acquisition, heartbeat renewal, concurrent acquisition, and lease expiration.
Every test passes.
But consider an execution scenario:
Worker A acquires a job and begins execution.
A long JVM pause causes its lease to expire.
Worker B acquires the same job and continues processing.
Worker A resumes and attempts to complete its original operation.
The lease-management tests may all be correct, yet the system could still permit a stale worker to modify state.
The problem is that the implementation and the tests share the same incomplete understanding of the requirement.
More generated tests may improve coverage without addressing the missing behavior.
This is not unique to AI. Human engineers make similar mistakes. However, autonomous coding workflows can carry the same assumption from implementation through validation without an independent challenge.
2. A Passing Test Is Evidence, Not Complete Proof
A green pipeline tells us that selected assertions passed under the conditions exercised.
It does not establish that every relevant system invariant holds.
For example:
Test | What it establishes | What it may miss |
Unit test | Expected component behavior | Cross-component failures |
Integration test | Selected system interactions | Untested failure sequences |
End-to-end test | Workflow behavior | Concurrency and recovery edge cases |
High code coverage | Code was exercised | Missing or weak assertions |
The strength of our correctness claim should not exceed the evidence supporting it.
This distinction becomes especially important in distributed systems, where correctness depends on execution ordering, concurrency, persistence, and recovery.
3. Verification Must Target the Guarantee
Let's revisit the scheduler.
Worker A acquires a lease with fencing token 41.
Its lease expires, and Worker B acquires token 42.
Worker B commits a protected write.
Worker A later resumes and attempts a write using token 41.
What should happen?
The protected resource must reject the stale write and preserve the newer committed state.
A meaningful validation scenario should reproduce this sequence using controlled execution checkpoints and verify the actual persisted result. Testing only that fencing tokens increase is insufficient. There is also an important limitation: fencing works only when the protected resource enforces the token ordering. It does not automatically guarantee exactly-once execution or prevent duplicate external effects.
The broader principle is straightforward: Validate the system guarantee, not merely the mechanism used to implement it.
4. Who Defines the Expected Result?
A test oracle determines whether an observed outcome is correct. But where does the expected result come from?
Suppose an AI agent implements a credit-limit calculation using an incorrect aggregation rule. It then generates tests using that same formula.
All tests pass.
The implementation and tests are consistent with each other, but potentially inconsistent with the approved business policy. For critical behavior, expected outcomes should have a basis independent of the implementation. This might include domain-approved decision tables, reference calculations, architectural invariants, or state-transition specifications.
That doesn't mean every test must be written by another person or agent.
It means the implementation should not become its own definition of correctness.
5. Who Validates the Validator?
Even an independently defined test can be ineffective.
How do we know a validation scenario can detect the failure it claims to detect?
One practical approach is to introduce a representative defect deliberately.
For the scheduler example:
Execute the stale-worker scenario against a correct implementation.
Disable the fencing check in an isolated test version.
Verify that the same scenario now fails.
Restore the check and confirm it passes.
This provides evidence that the test detects that particular violation. Other useful techniques include mutation testing, deterministic concurrency testing, fault injection, property-based testing, and model-based testing. None provides universal correctness guarantees. But together, they can strengthen confidence that our tests are capable of exposing the failures we care about.
6. The Verification Trust Boundary
Should an AI agent be allowed to write and execute tests?
Absolutely.
It should also be able to investigate failures, propose additional scenarios, and correct implementation defects. But should it have unrestricted authority to modify the criteria used to accept its own work? I don't think so.
For critical systems, I would separate three responsibilities:
Responsibility | Accountability |
Development | Agent implements changes and proposes tests |
Verification | Trusted infrastructure executes approved checks and records evidence |
Acceptance | Protected policies and authorized reviewers determine eligibility |
An agent may run validation through a CLI, MCP tool, or CI pipeline. However, it should not silently weaken assertions, remove mandatory tests, or alter approved requirements simply to make a pipeline green. These boundaries require enforceable permissions and workflow controls not just instructions in a prompt.
The ability to execute validation is not the same as the authority to define acceptance.
7. Make Validation Evidence Traceable
A test report stating "150 tests passed" is useful, but incomplete for a critical engineering decision.
We should also know which implementation revision was tested, which acceptance criteria applied, which scenarios ran, and whether any required checks were skipped.
For important changes, I would expect evidence linking:
Requirement → Scenario → Execution → Assertion → Result
The goal is not additional reporting overhead. It is ensuring that engineering confidence can be traced to observable behaviour. A reliable acceptance process should also treat missing or invalid mandatory evidence as an incomplete validation, not as success.
8. Start Small, Validate What Matters
Not every code change requires heavyweight independent verification. The effort should be proportional to architectural risk. A formatting change and a distributed ownership change do not require identical assurance. I would start with one critical invariant in an existing service. Define the expected behavior independently. Build a deterministic validation scenario. Introduce a known defect to verify that the test detects it. Then allow the AI agent to invoke the same test during implementation, while CI independently enforces the required result. Measure whether this reduces escaped defects and rework before expanding into a broader verification platform.
My Perspective
I don't believe we should prevent AI agents from testing their own code. That would limit one of their most useful capabilities. But I would not allow an implementation and its self-generated tests to become the sole basis for accepting a critical change.
There is a fundamental difference between two questions:
"Did the tests pass?"
And:
"Do these tests provide credible evidence for the guarantees we are claiming?"
The second requires sound specifications, meaningful test oracles, adequate scenarios, trustworthy execution, and traceable evidence. As AI coding agents become more autonomous, I believe verification independence and test adequacy deserve the same architectural attention as reliability, security, and scalability.
Let AI agents implement. Let them test. But make correctness independently demonstrable.
