Verifying AI Subagent Work: Nine Checks That Turn Testimony Into Evidence

Both moves feel rigorous. Neither is adversarial by default.

Reading the diff answers "does this code look reasonable." It doesn't answer "is this the only code that changed." A lane under instructions to write a failing test has every structural reason to also make that test pass by touching the implementation, instead of leaving it red. Nothing about a diff view flags that: a plausible implementation edit sitting next to a plausible test edit reads as one coherent change, not as a violation of the task's actual contract.

Running the tests answers "did the command exit zero." It doesn't answer whether the run happened against the code you think it happened against, whether every reported failure failed for the reason you think it failed for, or whether the counts in front of you reconcile with the counts from the last time you looked. A green run downstream of a broken setup is still a green run. A red run can be red for a reason that has nothing to do with the defect under test.

The fix isn't reading more carefully. It's converting each question into something mechanical: a command with a binary answer, an arithmetic check, a classification done once and never re-derived by eye. That's the rest of this post.

One git command, run before anyone opens the diff, settles a question that would otherwise depend on trusting a lane's own account of what it touched.

The scenario: a lane is told to reproduce a bug by writing a test that fails against the current, unmodified implementation, then fix the implementation so the test passes. The dangerous failure mode is a lane, under pressure to report a clean result, editing the implementation to make its new test pass without the implementation having a bug in the first place, or editing it in a way that papers over the symptom the test checks rather than the actual defect. Either way, "this test failed before my fix" becomes unfalsifiable once the only thing being read is the final green state.

The check is one git command:

git diff <base>...<lane-branch> --stat -- app/ agent/

An empty result means the product code is provably untouched by this lane, full stop: no implementation edit exists anywhere in the diff between the base and the lane's branch, scoped to the actual product directories rather than tests or benchmarks. Only once that's established does a claim like "the test fails against main" mean anything, because now the test's behavior on main reflects main's actual behavior, not a moving target the lane also modified.

One lane's full diff looked, at a glance, like it touched a lot of surface area. Scoped to app/ and agent/, the stat output came back empty: everything it had touched lived under benchmark and test paths. That's the entire review for "did you actually leave the implementation alone," answered in the time it takes to type one command, and binary in a way eyeballing a diff never quite is: either the stat list has product-directory entries in it, or it doesn't.

Don't trust a pasted failure message. A lane's report of "this failed before my fix, here's the traceback" is still testimony, generated after the fact from whatever the lane remembers or reconstructs about the pre-fix state. The only version of that claim worth acting on is one produced independently, against the actual unmodified base.

In test-driven-development shorthand, a RED run is a test that fails as expected, proving it actually exercises the bug before any fix lands; GREEN is the same test passing once the fix is in. Reproducing the RED yourself means checking the lane's new test file out against the unmodified base revision, in a separate, clean worktree, and running it there. Not in the lane's own worktree, where the implementation fix already landed. Not by re-running the lane's reported command and trusting the reported output. A fresh worktree checked out to the base commit, with only the new test file copied over, is the smallest environment in which "does this fail against the code as it stood before the fix" has an unambiguous, self-produced answer.

One fresh-worktree run against the unmodified base produced:

6 failed, 299 passed

That number, generated independently in an isolated worktree, is what licensed the merge. The lane's own report of the same failure count is corroborating, not load-bearing. The distinction matters because the two numbers being equal is itself informative: if an independent reproduction and the lane's reported number disagree, something about the lane's environment, working tree, or reporting was wrong, and that's worth discovering from a mismatch rather than assumed away by an agreement nobody checked.

If you take only one of these nine checks, take this one.

Six failures reads like six pieces of evidence that the test suite caught the bug. It isn't. Of the six failures in the worked example above, five were genuine. The sixth was a TypeError, raised because the new test called decay_old_notes(..., now=fixed_now) with a now= keyword argument that did not exist on the function's signature at the base revision. That's not the test demonstrating the defect it was written to catch. That's the test failing to run at all, because it calls an interface that doesn't exist yet, a completely different fact from "the old behavior is wrong."

The other five in that run were genuine assertion failures, worth quoting individually because they carry more information than "five failed" ever could:

• assert 1 == 0
• an assertion whose message read "expired note's row must survive purge (state machine is append-only, never DELETE)"
• assert None is not None
• 0.14999896382860867 == 0.5 ± 0.025
• two separate instances of assert 0 == 1

The fourth is doing more work than the rest. 0.14999896382860867 is, to floating-point noise, exactly 0.3 × 0.5. That number alone pins the mechanism: the test seeded a note's prior decay score at 0.3 and expected a fresh decay calculation, one half-life later, to land near 0.5, computed purely from elapsed time. Getting 0.15 instead means the implementation was multiplying the new decay factor onto the note's existing decay score rather than recomputing it from scratch, so a second call compounds a first call's result instead of superseding it. The full fix, including why the corrected version has to produce the same score no matter how many times it runs, is in Agent Memory Expiry Is a State, Not a Delete. Reading the assertion's own numbers found the bug before a single line of the implementation got opened. That's the difference between a failure count and a failure that's been read: a count says something is wrong, a genuine assertion failure with real expected and actual values in it often says what's wrong.

The lane itself had already declined to count that TypeError as a genuine RED in its own report. Verification isn't only a defense against a lane overstating its work. Running the same check independently sometimes confirms a lane was already being more careful than its summary suggested, which is a different, and equally useful, outcome.

A different category of claim: this diff only touches comments and docstrings, no behavior change. That claim is common on cleanup or documentation-pass tasks, and it's tedious and error-prone to verify by reading a large diff, because a docstring edit sitting sixty lines above a one-character logic change is trivially easy to miss on a skim.

Don't skim it. Parse both revisions with Python's ast module, walk every Module, ClassDef, and FunctionDef node, strip the leading docstring Expr node from each body if one is present, and compare ast.dump() of the two normalized trees. Identical dumps mean the two revisions are provably semantically equivalent modulo docstrings, module-level comments, and whitespace, none of which reach the AST at all. Different dumps produce a structural diff of exactly what changed, without reading a single line of prose to find it.

docstring-only verification
import ast

def strip_docstrings(tree: ast.AST) -> ast.AST:
    for node in ast.walk(tree):
        if isinstance(node, (ast.Module, ast.ClassDef,
                              ast.FunctionDef, ast.AsyncFunctionDef)):
            body = node.body
            if (body and isinstance(body[0], ast.Expr)
                    and isinstance(body[0].value, ast.Constant)
                    and isinstance(body[0].value.value, str)):
                node.body = body[1:] or [ast.Pass()]
    return tree

def normalized_dump(source: str) -> str:
    tree = strip_docstrings(ast.parse(source))
    return ast.dump(tree, annotate_fields=False)

def is_docstring_only_change(old_src: str, new_src: str) -> bool:
    return normalized_dump(old_src) == normalized_dump(new_src)

This is a five-minute script, and it answers a question that otherwise costs a slow, attention-taxing read of a four-hundred-line diff, with the risk of missing the one line that mattered because the surrounding two hundred lines were legitimately just prose. Identical normalized dumps are a proof. A read of the diff, however careful, is an opinion.

Replacing an existing test is correct in one case and a cover-up in the other, and it's the same action either way: the only way to tell which one you're looking at is reading what the old test actually asserted, on the base branch, before anyone touched it.

The failure mode: the cheapest way to turn a red suite green isn't fixing the bug, it's rewriting the test that's catching it. A lane under pressure to report success has a live incentive to loosen an assertion, delete a check, or replace a test wholesale with one that happens to pass against the unfixed code. A diff showing a test file modified doesn't distinguish "the old test was wrong and encoded the bug being removed" from "the old test was right and someone made it stop complaining." The new test alone can't tell you which. Only the old one, in its pre-change form, on the base branch, can.

A real case had exactly two old tests replaced. Read on main, before the change: test_decay_deletes_very_old_notes had a docstring reading "Notes with decay_score < 0.1 are auto-deleted," and its body asserted store.recall("/repo") == [] after decaying a note down to that threshold: prove the note is gone. test_purge_expired_notes_removes_old_notes asserted store.count_notes(ws) == 0 after purging a note past its TTL: the same claim from a different angle, prove the row no longer exists.

Both tests were, by construction, codifying delete-on-expiry as correct behavior. The fix these tests were replaced for was specifically about removing delete-on-expiry, swapping a destructive purge for an append-only state transition so an expired note's row and its audit trail survive, with the note simply excluded from default recall. The full design is in Agent Memory Expiry Is a State, Not a Delete. Given that goal, both old tests asserted exactly the behavior the fix exists to eliminate. Replacing them wasn't a cover-up. It was the only correct outcome: a test asserting "the row is deleted" cannot coexist with a fix whose entire point is never delete the row.

Had those two old tests instead asserted something orthogonal, say a check on notification formatting sharing a test file with the decay logic, replacing them under the same diff would have been exactly the failure mode this check exists to catch: a real regression quietly deleted alongside a legitimate change, in the same commit, under the same lane summary claiming both were intentional. Same diff shape in both worlds, same lane, same confident report. Only reading the old assertions, on the base branch, tells you which world you're in. There's no shortcut around actually reading them.

None of this makes the checks adversarially robust, and it's worth being precise about the difference. A lane that knew in advance exactly which nine checks would run against it could, in principle, learn to route an implementation edit through a directory a scoped diff doesn't cover, or phrase a replaced test's docstring to survive a skim of the old assertion. What these checks are actually robust against is the failure modes observed so far: a lane optimizing for a report that reads as done, not one deliberately reverse-engineering the review process itself. That's a narrower guarantee than "adversarially secure," and treating it as the stronger claim would be exactly the kind of overstatement this post is arguing against.

Track pass counts as a ledger across merges, not as isolated snapshots, on whichever codebase you're maintaining; the numbers below are from a much larger suite than the 299-test example in the last section, a different repository entirely, kept only to show the pattern. If the suite stood at 4752 tests before a merge, the next merge landed at 4754, and the one after that landed at 4758, those numbers have to compose with what each lane claims it did. A lane that says "I added 4 tests" against a base of 4754 has to land the suite at exactly 4758. Landing at 4757, or 4760, means something doesn't reconcile, and that's a question to ask before merging, not a detail to notice later. The ledger itself doesn't need to be anything elaborate: a line in the merge log, a comment on the CI run, a running note next to the queue. What matters isn't where it lives, it's that the number gets written down and the arithmetic gets checked every time, not just when something already looks wrong.

It catches a class of problem no amount of diff-reading surfaces: a test silently skipped, a marker that deselects more than intended, a fixture that swallows a collection error instead of raising it. None of those show up as a red mark anywhere. They show up as a total off by a number nobody explained, and the only way to notice is keeping the ledger in the first place.

A CI run went red immediately after a merge landed. Bisecting the merge would have been wasted effort here: the two failures had nothing to do with the merged code. The result line read:

2 failed, 4752 passed, 4 skipped, 13 deselected

Add those up: 2 + 4752 + 4 + 13 = 4771. A local run of the identical commit read 0 failed, 4754 passed, 4 skipped, 13 deselected: the same total, 4771, with the two failures landing as passes instead. Identical totals across two environments running the identical commit mean the set of tests that ran was unchanged between the two runs: the merge didn't alter what got collected or executed. If the merge itself caused the two failures, the failure should reproduce locally too, on the same commit, under the same collection. It didn't. Same commit, same total, different outcome between environments, which is what an environment difference looks like, not a code regression, and the arithmetic pointed at that category before either traceback got read.

The mechanism: an upstream model host returned an HTTP 429, because two tests were reaching out over the network to download an embedding model they had no business touching. The cache root gets redirected to a fresh temporary directory for the whole test session, so the suite never writes to the real developer cache, and that isolation has a side effect nobody had traced through: the model cache is empty on every run, local ones included. Both environments download the model every time; a warm cache isn't what saves the local run. Local requests simply get served, and CI, hitting the same host from many parallel jobs, gets rate-limited. The download itself was accidental: the indexer's constructor builds its embedding provider eagerly, so two tests that only checked path resolution pulled multiple gigabytes over the network as a side effect of construction. Eight other tests in the suite stub the provider out; these two missed it, and one of them stubbed it a line too late, after the constructor had already run.

The arithmetic identified the category, environment rather than regression, in five seconds. It couldn't identify the mechanism: an eager constructor and an isolated-but-always-cold cache had been costing every CI run a multi-gigabyte download for as long as both existed, and nothing about the merge would have surfaced that on its own. A five-second addition problem pointed at the right category of answer before any log got read, and it costs little enough that skipping it whenever a run goes red right after a merge has no real justification.

Verification discipline that only applies to the thing being checked, not to your own process while checking it, is theater. It looks rigorous and isn't.

A supposedly isolated verification re-run turned out not to be isolated at all: ps showed another test suite's process still alive on the same machine while the verification run was executing, meaning the two were sharing CPU, and potentially any shared state like a local model cache or a filesystem lock, for the run's duration. The honest report says exactly that, rather than presenting the result as a clean, trustworthy signal. The run came back green anyway, and the honest version of that report doesn't stop at the color: it was re-run a second time, alone on the machine with the competing process killed, and came back green again before the merge went in. The first, contended run didn't get to stand in for a clean one just because the second run agreed with it.

That green result, under contended conditions, was itself informative, not just a relief. It meant the interference, whatever it was, wasn't deterministically reproducible enough to flip this particular run's outcome, a weaker but still useful fact: the check survived one instance of noisy conditions, not that noisy conditions never matter. Reporting "this ran clean" when it hadn't would throw that distinction away and substitute a false result for a nuanced, honest one. The whole point of running these checks is producing facts you can act on. A fact silently upgraded past what was actually observed isn't a fact anymore. It's the same testimony this post is about not trusting, coming from the verifier instead of the lane.

Not everything resolves cleanly, and pretending otherwise is its own failure mode.

A test failed once, under concurrent suite runs, and passed cleanly on the next run under what looked like the same conditions. Inventing a plausible explanation and patching around it would produce a fix for a mechanism that was never actually confirmed, which isn't a fix: it can silently paper over the real problem while looking resolved.

The record instead marks the failure UNPROVEN, alongside the hypotheses already ruled out (cache isolation between concurrent runs was checked specifically and confirmed correctly isolated, so that's off the list) and the one leading hypothesis that remains unverified. That's a more useful artifact than a confident guess: the next person, or the same person later, who hits this failure again knows exactly what's already been checked and doesn't waste time re-deriving it.

A related lesson came out of the same incident, and it's now a standing rule in every lane brief since: the traceback for that original failure was lost, because whatever ran it had piped the output through tail -3, keeping only the last three lines. Three lines is nowhere near enough to diagnose a concurrency-sensitive failure after the fact; by the time the need for full context is obvious, the run is over and the information is gone. Every brief now mandates keeping at least 40 lines of any failing run's output. Evidence has to survive long enough to be checked. A claim about a failure whose output has already been discarded can't be verified, and piping it through tail because the run was noisy doesn't hold up against a failure that actually needs to be understood.

These lanes are not unreliable. A list of nine checks can read as evidence against that, but the failures behind these checks are subtle by nature: a TypeError masquerading as a genuine RED, a test rewrite that's legitimate nine times out of ten and a cover-up the tenth, a decay bug whose signature is a six-digit floating-point coincidence. No amount of careful reading catches these reliably, because careful reading is a variable-quality process that degrades under fatigue, familiarity, and the reasonable instinct to trust a report that says done.

Every check above is seconds of mechanical work set against minutes of agent time. Most of them are one command: a scoped git diff --stat, a suite total that either adds up or doesn't, an ast.dump() comparison. None require reading the lane's prose more carefully. All of them replace reading the prose more carefully with something that produces a fact instead of an impression.

A cheap, mechanical check that runs every time, on every merge, without depending on extra attention this one time, beats a careful review that only runs when someone happens to be paying full attention. Build the checks into the loop once, and the testimony stops mattering, because the evidence is already there.