Done Is Not Verified: Evidence Receipts for AI Coding Agents
An AI coding agent saying "fixed" is a claim, not a result. How we make agents carry evidence, declare boundaries, and earn a verification receipt before work counts as done.
Abdul Hamid Achik
The Claim Problem
Ask a capable coding agent to fix a bug and you’ll get a confident answer within minutes. The function compiles, a test passes, and the summary says the issue is resolved.
That summary is a claim. It may be true, or it may be a guess that happened to survive one command.
When the claim is wrong, the cause is seldom the model’s ability to write code. The agent loses discipline across a multi-step task:
- Read the bug report.
- Run a broad search and open a few files.
- Infer a likely cause.
- Edit several files.
- Run one command.
- Treat partial success as proof.
- Lose the trail of why each decision was made.
Each step looks reasonable on its own. Together they produce a codebase where nobody, human or agent, can say which changes were proven and which were hopeful. This is the same gap we see in vibe-coded MVPs: the code works in the demo, and nobody can tell you why it should keep working.
Status Messages Are Not Evidence
The fix is not a better prompt. Telling a model to “be careful” or “double-check your work” changes the tone of its summary, not the structure of its process.
What changes outcomes is making certain steps impossible to skip. We built cortex, an open-source, local-first runtime that sits between the model and its tools, around that idea. The model stays the planner and the author. The runtime enforces a phase machine:
orient → investigate → form hypotheses → declare a boundary → change → verify → preserve evidence
The point is not the tool. The point is four artifacts that replace “done!” with something a reviewer can check. You can adopt them with or without cortex.
Four Artifacts Instead of a Status Message
1. An Acceptance Contract Written Before the Work
Before the agent touches anything, the task records what success means in user-visible terms. Not “fix the redirect bug” but “a login started at checkout returns to checkout.”
The contract is immutable once the work starts. That matters because agents drift. Halfway through, a hard bug quietly becomes an easier bug, and the summary declares victory over the easier one. A contract fixed at the start makes that substitution visible.
2. Evidence With Provenance
Search results are candidates, not facts. When the agent searches the codebase, each hit is recorded with where it came from, and weak hits are filtered out. If a whole round of searching finds nothing strong, the honest output is “no strong candidates,” not three plausible-looking files.
Provenance pays off when you review an agent’s work after the fact. With provenance, you can see that the agent concluded “the token is dropped in the callback” because it read callback.ts, not because a filename looked related.
3. A Hypothesis With a Disproof Path
A plan is only accepted if it says how it could be wrong. “The return URL is dropped before the OAuth callback” is a hypothesis. “Run the login-from-checkout browser flow; if it still lands on the dashboard, the hypothesis is false” is the disproof path.
Plans without a disproof path are rejected outright. This is the single most effective rule we have found. It forces the agent to name the observation that would prove it wrong before it starts editing, which is exactly the step that gets skipped when someone is in a hurry.
The plan also declares a change boundary: the files and symbols the agent expects to touch. Before editing, the agent claims a lease on that boundary. Leases expire and prevent two agents, or an agent and a human, from writing over each other.
4. A Verification Receipt Per Claim
Verification binds a specific claim to a specific surface and a specific check. “Login started at checkout returns to checkout” is verified on the browser surface by a named end-to-end spec, at a specific git revision.
The result is a receipt: this claim, this check, this revision, this outcome. If the diff touched files outside the declared boundary, that scope drift is reported next to the receipt instead of disappearing into a large commit.
When Verification Refuses to Pass
Every task ends in one of four assessments: verified, partial, failed, or unverified. Normal completion requires verified. Anything else can still be recorded, but only with an explicit acknowledgment, so a partial fix is preserved as partial instead of being rounded up to done.
Expect more tasks to end as partial than you’re used to. That isn’t a regression: those tasks were partial before, and the old summaries rounded them up.
Two other safeguards matter in practice:
- Commands need approval from outside the repository. A repository’s own configuration cannot approve the commands it asks the agent to run. The trusted launcher has to opt in. Otherwise a malicious or careless config file becomes a way to execute arbitrary code.
- Missing tools are reported, not imagined. When a specialist tool is not installed, the adapter degrades and says so. It never fabricates output for a tool that did not run.
What Evidence Does Not Prove
Receipts are only as good as the checks behind them. A verification receipt bound to a weak test proves that the weak test passes. If your test suite never exercises the behavior a customer cares about, a perfect evidence trail will faithfully record that you checked the wrong thing.
That is why the acceptance contract is phrased in user-visible behavior, and why browser and end-to-end checks carry more weight than a unit test that mocks the interesting part. We made the same argument from a different angle in Looks Correct Is Not Is Correct: running the thing beats reading the thing.
Evidence also does not replace judgment about what to build. It tells you whether a change did what it claimed. It does not tell you whether the claim was worth making.
How to Apply This Without Any New Tools
You can get most of the value with a pull request template and a review habit:
- Write the acceptance criterion first, in user-visible terms, before any code changes.
- Require a disproof path in the plan: the observation that would show the fix is wrong.
- Declare the change boundary, and treat edits outside it as a review flag, not a detail.
- Attach evidence per claim: which check, at which commit, with what result.
- Allow “partial” as a final state. A partial fix that is labeled honestly is cheaper than a full fix that is not.
If agents write a large share of your code, these rules are the difference between a codebase you can reason about and one you can only hope about. If you are already living with the second kind, our guide to maintaining vibe-coded projects covers the audit side, and the cortex source is on GitHub.
Abdul Hamid Achik
Founder and lead engineer at The Lacanians. Abdul builds production software, developer tools, and local-first systems from Guadalajara for teams worldwide.