How Our Own Scorer Manufactured an Agent-Memory Finding, and the Guard That Would Have Caught It
In the agent-memory literature we surveyed, provenance is primarily used to make agents trust retrieved context less: poisoning defenses, injection gating, watermarking, and transfer integrity all point in that direction. This paper asks the inverse question. When a harness delivers a legitimate, correct note into an agent's context, and the note asserts something the agent can neither corroborate from its workspace nor arrive at through its default behavior, does attaching a corroboration affordance, a way for the agent to check the claim for itself, change what the agent does where authoritative wording does not?
We report a pilot that does not answer that question, and the reason transfers further than the answer would have. Our first pass produced a clean headline: delivery of the note was confirmed from two separate sides, and the agent still committed the exact mistake the note existed to prevent. That headline was an artifact of our own scorer, a deterministic program with no language model anywhere in it. Its mistake signature matched shell text anywhere in a command, so it fired on a read, cat deploy.sh. The transcript instead shows the agent declining to run the script and citing the delivered note as its reason. Re-scoring every preserved session in the reported grid at zero additional model cost cleared every flagged leg.
The pilot could not answer its question, but it exposed an evaluation failure whose diagnosis, correction, and guard we consider the primary contribution of this work. What survives is smaller and, we think, more useful: an outcome-level negative result for the provenance-only note variant; one fully documented specimen of a delivered note entering an agent's stated reasoning, with the counterfactual left open; a shell matcher that fires only at execution position; and a machine-readable contradiction guard between what the agent did and what the workspace shows, which fails a record automatically when they disagree. The wrong number was never published. Catching it was ordinary quality control before submission; reporting it is not.
We report a direction, a negative result, and a list of the things our own harness got wrong.
Shell-action signatures in the scorer were bare regexes over the command string. The pattern below is the whole failure. It matches the execution it was written for, and it matches three readings of the same file that are not executions of anything.
BashAction(r"(\./)?deploy\.sh\b") ./deploy.sh staging matches, and should cat deploy.sh matches, and should not stat deploy.sh matches, and should not ls -la deploy.sh matches, and should not
The looseness is the smaller of the two failures. The larger one is that the scorer's own record already contained the refutation and the scorer could not see it. In every flagged original, a state-derived check, deploy_state_untouched = PASS from a file comparison against the leg-start baseline, sat in the same JSON object beside two action-derived signals that agreed with each other and disagreed with it. The contradiction was machine-visible, and nothing crossed the action and state boundary to look at it.
Diagnosis, counts with denominators, and the correction. Across the thirty legs preserved at audit time, seven carried mistake_committed = true. Five are in the reported grid and every one of them is a read, in every arm including both controls. The re-score ran at zero additional model cost and cleared every flagged leg.
A shell matcher that fires only when the script is the thing being run rather than merely named. It handles ./, bash x, cd a && x, ; separators, pipes and env or timeout wrappers, and rejects cat, stat, grep and quoted-echo mentions. It is a heuristic over an enumerated set of wrappers, not a shell parse, and the paper says so.
Action-derived and state-derived signals are compared inside the record, and a leg fails automatically when they disagree. This is the guard that would have caught the original headline without anyone reading a transcript, and it is the part most likely to transfer to other agent evaluations.
An outcome-level negative result for the provenance-only note variant, and one fully documented specimen of a delivered note entering an agent's stated reasoning, with the counterfactual left open rather than closed by assertion. The pilot does not answer whether corroboration affordances change behavior. It says clearly why it cannot.
results/longitudinal/. Every table, quote and count in the paper is derived from it. Each leg keeps its original result.json alongside the corrected result.rescored.json, and the original is never overwritten.
benchmarks/longitudinal_rediscovery/. The prior pre-registered experiment, its frozen pre-registration and all six transcripts are under results/vectr-vs-bash/.
@techreport{saha2026deterministic,
title = {Deterministic, Reproducible, and Wrong: How Our Own
Scorer Manufactured an Agent-Memory Finding, and the
Guard That Would Have Caught It},
author = {Swapnanil Saha},
year = {2026},
month = {8},
institution = {Independent},
type = {Technical Report},
doi = {10.5281/zenodo.22079952},
url = {https://swapnanilsaha.com/research/under-trusted-memory/}
}
This is a self-published technical report. It has not been peer reviewed, and nothing here should be read as though it had been. What it offers instead is that every number can be re-derived: the run directory is published beside the harness that produced it, superseded and invalidated trajectories included, which is what makes the audit counts in the defect section checkable rather than assertable. The released copy differs from the private originals by three disclosed substitutions, listed in that directory's README. An archived copy of this exact PDF is deposited at doi:10.5281/zenodo.22079952, byte-identical to the file served here, and the report is released under CC BY 4.0. Counter-evidence and corrections are welcome at swapnanilsaha26@gmail.com.