Your Eval Is Leaking the Answer: Paths, Fixtures, Residue, and Your Own Git Identity
A control session that should have known nothing named the hidden mechanism out loud. It had read the answer off its own working directory. Five leak classes, all found in one harness, with the fix for each.
The sentence that ruined my week was: "this environment is explicitly testing reconciler behavior." It appeared in a transcript from an agent that had no way of knowing that.
I have been building a benchmark that measures whether a persistent memory layer makes a coding agent's second, third and fourth session on the same codebase cheaper than its first. The design turns on a control group: one arm of the experiment runs with no memory at all, and its cost is the number every other arm is measured against. In the scenario in question, the hidden fact was that deploying to staging by running ./deploy.sh looks like it works and gets silently reverted a few minutes later by a reconciler process, leaving no trace anywhere in the repository. The word "reconciler" appears zero times in that workspace. I checked, because I wrote it.
The no-memory agent predicted, in its second session, that its change would "get flagged as drift and reverted." It was right, and it had no business being right. The answer was in its shell prompt: the directory the harness had put it in was named deploy_reverted_by_reconciler-none-none-s0.
Scoring an agent is the easy half of building an eval. The hard half is proving that the thing you are measuring could not have arrived by any other road. This post is about five roads I did not know were open, four of which I paved myself.
What a Memory Eval Is Actually Claiming
Start with the claim, because the leak classes only make sense once you see what they break.
A coding agent is a language model wired to tools: it reads files, runs commands, edits code, and stops when it thinks the task is done. A session is one such run, and every session starts from nothing. Whatever the agent worked out yesterday about your build system is gone; the model does not carry state between conversations. A memory layer is any mechanism that hands facts from an earlier session back to a later one, whether that is a notes file, a retrieval tool, or a startup injection.
The claim under test is a claim about waste. Without memory, session two re-derives what session one already knew: it re-reads the same files, re-runs the same searches, and sometimes re-makes the same mistake. With memory, it does not. My benchmark measures the difference in turns, tool calls, tokens and dollars.
Notice that the quantity of interest is a difference. That single fact governs everything downstream. To get it you run the same scenario twice under different conditions, which in experimental language are arms. The arm with no memory is the control arm. A multi-session run of one scenario is a trajectory, and each session inside it is a leg.
The scenarios themselves split into two kinds, and the distinction turns out to matter enormously for leak hunting.
- Corroborable facts. The agent could work the fact out from the repository if it looked hard enough. Releases go out through continuous integration and there are no upload tokens on the machine: annoying to discover, but discoverable. Memory saves you the discovery cost.
- Uncorroborable facts. Nothing in the repository states or implies the fact. The deploy script that silently gets reverted is the canonical example: the script exits zero, the reconciler runs somewhere else, and the revert leaves no artifact. The fact enters the agent's world exactly once, in the first session's prompt, spoken by a human. After that, memory is the only channel that can carry it forward.
Uncorroborable scenarios are where a memory layer either earns its keep or does not, because there is no fallback. They are also the ones a leak destroys most completely, because "the agent cannot re-derive this" is not a soft assumption there. It is the entire construct.
That word is the one the field uses. What follows is not a bug in the code and not a statistical problem; it is a construct validity failure, which is the unglamorous category where a number is computed correctly and means something other than its label.
Which gives the invariant this whole post hangs on.
An eval measures memory only if the fact being remembered cannot be re-derived from the environment. Not "is unlikely to be re-derived." Cannot.
The moment there is any path from the environment to the fact, you are no longer measuring memory. You are measuring the agent's ability to notice that path, which is a different and much less interesting question.
One more thing, in the interest of you knowing where I am standing. The memory layer under test is a tool I built, which I have written about at length. Grading your own homework has a characteristic failure mode, and it is not the one people expect. You do not fake the treatment number. You just never look very hard at the baseline, because the baseline is behaving the way you hoped it would. Which is how this leak survived a full campaign.
The Observable Surface Is Bigger Than the Workspace
When I built the scenarios I was careful about the files. Every fixture got read twice. There is an automated check that no distinctive phrase from the fact sentence appears in a file the agent can open. I was, in a narrow sense, rigorous.
I was also thinking about the wrong object. I had been guarding "the workspace" when the thing that needed guarding was "everything the agent can observe," and those are not close to the same set.
You are invigilating a closed-book exam. You collect the textbooks, the notes, the phones. The room is clean. Then you hand out the paper, and the header on every page reads: Trigonometry, Section 4: the Law of Cosines.
You did not leave the book open. You printed the chapter title on the answer sheet, and you did it while congratulating yourself on how thoroughly you had cleared the desks. The candidate is not cheating when they use it. Reading the paper in front of you is the job.
For a coding agent, the context window is assembled from far more than the prompt. The working directory is in there, because it turns up in the shell prompt and in every absolute path any tool prints back. So are environment variables, the first time anything runs env or a command dies with a config error. So is whatever startup payload the harness or the command-line tool injects before turn one, the text of every file the agent opens, the output of every command it runs, and the filesystem it inherited from last session.
Every one of those is a string. To the model a string is a string; there is no metadata channel and no privileged tier. There is only what arrived.
Before the war stories, the arithmetic, because it decides how much you should care.
Normalise the per-session saving so an agent that has the fact scores 1 and an agent that does not scores 0. The true gap between arms is then 1 by construction. Writing E[·] for the average over many trajectories:
E[Y_memory] = 1 E[Y_control] = q · 1 + (1 - q) · 0 = q measured gap = 1 - qRead q as the fraction of the benefit the side channel hands the control arm, averaged over runs. Fire rate and strength collapse into the same number: a leak that fires half the time and gives away everything, and a leak that fires always and gives away half, both come out at q = 0.5. A path leak is usually the second kind, a strong hint rather than the full sentence, which is why measuring it is so unpleasant.
The consequential part is what happens next. Random noise shrinks as you add trajectories: the standard error goes as σ / √n. Bias does not shrink at all. Spending more compute narrows your confidence interval around a number that was wrong to begin with, so past a certain sample size the interval stops containing the truth entirely. The threshold is where the interval's half-width no longer covers the shortfall:
q* = 1.96 · σ / √nAt σ = 0.5 and n = 64 trajectories, q* is about 0.12. A leak firing one time in eight is enough to put the real effect outside the interval you report, and every trajectory you add after that makes the exclusion more emphatic.
One honesty note on σ. If the outcome really were the coin flip above, σ would be √(q(1-q)) and would move with the slider. In the actual harness the outcome is a continuous cost saving in turns and tokens, and its spread is dominated by how much sessions vary in difficulty, not by the leak. Holding σ at 0.5 keeps the demo about the bias term instead of tangling it with a variance term that has nothing to do with the point.
That is the part I want to be blunt about, because it inverts the usual instinct. When a result looks noisy, you run more seeds. When a result is leaked, running more seeds is the worst thing you can do: it converts an uncertain wrong answer into a confident wrong answer, and it costs you money to do it.
Set how often the control arm recovers the fact through a side channel, and how many trajectories you are willing to pay for. Watch the reported interval tighten around a number that is not the truth.
Leak One: The Working Directory Told the Agent the Answer
Every trajectory in the harness has an identifier built from the four things that define it: which scenario is being run, which arm it belongs to, which note variant was used (the same remembered fact can be phrased tersely or at length, and phrasing turns out to matter), and the seed, which is just the repeat number so the same cell can be run several times. Written out, that is deploy_reverted_by_reconciler-none-none-s0. It is a good identifier. Unique, sorts sensibly, and when you are staring at a results file at two in the morning you can tell at a glance which cell you are looking at.
So of course it was also the directory name. The runs directory held one folder per trajectory, each named after its id, and inside each folder sat workspace/, which is the directory the agent is spawned into. Which means the agent's cwd was:
<runs-dir>/deploy_reverted_by_reconciler-none-none-s0/workspaceRead that as a language model. You are dropped into a repository. Before you have opened a single file you have been told the scenario ("deploy reverted by reconciler"), your experimental condition ("none", meaning no memory), and the seed. You did not have to be clever. You had to read your own prompt string.
And the model was clever anyway. The transcript reasoned that the environment was "explicitly testing reconciler behavior," then went further and predicted the specific consequence: that a direct deploy would "get flagged as drift and reverted." That is not a lucky guess about a generic reconciler. It is the scenario's hidden fact, reconstructed from four words in a path.
Not one trajectory. The control arm is the baseline every other number in the campaign is measured against, so a control that knows the answer does not produce one bad cell. It drags the baseline toward the treatment arm across every scenario sharing the flaw, and flattens every contrast built on top of it. There is no partial credit and no post-hoc salvage. The runs are firewood.
It is also worse than one label, because an agent is not confined to its own cwd. It has a shell. cd .. costs one tool call, and from the runs directory it can list every sibling trajectory in the campaign, at which point it is not reading a label but an experimental design: the arm names, the variant names, how many conditions exist, which ones have already run. That reach is the argument I find most persuasive for opaque naming. Move the agent somewhere neutral and the campaign directory is still one level up and still readable.
What makes this class nasty is that nothing was broken. The id was correct. The directory layout was correct. The workspace contents were correct and passed their own leak test. The bug lived in the composition, in the fact that a naming convention met a working directory, and no unit test in the world has an opinion about that. It only exists when you look at the system from where the agent sits.
The fix: make the name carry no signal
The obvious move is to stop using the id as a path. The subtlety is that you cannot simply delete it, because the harness still needs to know which cell is which, and there are campaigns already on disk under the old names.
def _opaque_run_dir_name(label: str) -> str:
"""Deterministic, non-reversible-by-inspection directory name."""
return "run-" + hashlib.sha256(label.encode("utf-8")).hexdigest()[:16]
def resolve_traj_dir(runs_dir: Path, trajectory_id: str) -> Path:
legacy = runs_dir / trajectory_id
if legacy.is_dir():
return legacy # pre-fix campaign: reuse as-is, never rename
return runs_dir / _opaque_run_dir_name(trajectory_id)Why a cryptographic hash for a problem with no attacker in it? Because the input space is tiny. Six scenarios times five arms times three note variants times a handful of seeds is a few hundred strings, and against a space that small, anything reversible or weak (base64, a rotation, a short checksum) leaks nearly as much as the plain slug. You can enumerate it by hand. I wanted output that is not readable as anything at all.
A security person would object to that sentence, and they would be right, so let me be exact about the threat model. This is obscurity, not secrecy. The harness source sits in the same repository, and anyone holding it can recompute the whole mapping in about a second. Fine, because the adversary here is not a person with a laptop. It is a pattern matcher reading its own shell prompt. A model that sees run-c7586c216fb01d55 gets nothing from it: inverting a hash means finding a preimage, and no amount of next-token prediction does that. The goal is a string that carries no signal to a reader operating in context. Defeating someone who has the code was never on the list.
Sixteen hex characters is sixty-four bits. At a few hundred trajectories the birthday-bound collision probability is somewhere around one in a quadrillion, and a collision would be loud rather than subtle, since two trajectories would try to write into one directory and the second would trip the runner's own state checks. Sixty-four characters would have been no safer and considerably more annoying to paste into a bug report.
The legacy branch is there because old campaigns are read-only history. Renaming their directories would invalidate every path already written into their result files, and the reason I keep them at all is that they are evidence. So the resolver looks for the literal old directory first and only mints an opaque name when there is not one. Both layouts coexist under a single runs directory. Every consumer of a trajectory id (the runner, the reporter, the analysis pass) goes through this one function, which is the part that matters: two code paths that compute a directory name independently will disagree eventually, and they will do it quietly.
The part I would carry to any other harness is not the hash. It is where the readable id went.
The human-readable id did not disappear when it stopped being a path. It became a field inside state.json and the results file: content that the analysis reads and the agent never opens.
Same string, different channel, opposite consequence. Metadata the agent can observe is prompt. Metadata sitting in a file only your tooling touches is data. Almost every fix in this post is that one move applied somewhere new.
Type any trajectory identifier and see which of its tokens survive into the string the agent can read. The hash is computed in your browser with the same SHA-256 the harness uses, so the directory name you get here is the one the runner would create.
Leak Two: The Session Bootstrap Carried the Same Path
Fixing a directory name takes ten minutes. The useful work was the hour after, spent asking where else that path went.
It went into the session bootstrap. Modern agent command-line tools let you run code at session start, and to do that they hand your code a small JSON payload describing the session that is beginning. My harness builds that payload itself during a preflight check, so that it can verify the startup path works before spending money on a real session. It looked like this:
{
"session_id": "hookpreflight-deploy_reverted_by_reconciler-k2",
"transcript_path": ".../hook-preflight-transcript.jsonl",
"cwd": "<runs-dir>/deploy_reverted_by_reconciler-none-none-s0/workspace",
"hook_event_name": "SessionStart",
"source": "startup"
}Two separate leaks in five lines. The cwd field is the path leak riding a second pipe, and it got fixed for free because it derived its value from the same function the directory fix changed. That was luck as much as design. Had I hardcoded the path in one of those two places, I would have fixed half a leak and felt finished.
The session_id is the more interesting one, because nothing fixed it for free. I named it that way for me. When six sessions are running and one of them wedges, a session id that tells you the scenario and the leg number is worth its weight in coffee. It is a debugging affordance, and debugging affordances are descriptive by definition, because a session id that reads a41f helps nobody at 3am.
That is the trap. Startup metadata is the easiest thing in a harness to name descriptively, precisely because nobody classifies it as content. It feels like plumbing. It is plumbing that empties into the context window.
Would you be comfortable pasting this field's value directly into the agent's first user message?
If yes, keep it. If the question makes you wince, the field does not belong in a startup payload either, because those are the same channel with different ceremony. A session id, a transcript filename, a config label and a prompt are, from inside the model, four strings that arrived at the same time.
The generalisation: anything your harness hands the agent before turn one is part of the environment. That includes startup payloads, injected system context, the contents of any config file the tool auto-loads from the working directory, and the tool's own banner if it echoes paths. Enumerate them. I found mine by grepping the harness for the scenario slug and reading every hit, which took twenty minutes and is the highest-value twenty minutes I have spent on this project.
Leak Three: Fixtures That Narrate the Test
The first two leaks were accidents of plumbing. This one I authored on purpose, with care, and still got wrong, which makes it the one I think about most.
A scenario is a workspace: a deploy script, a config file, a couple of source files, a README, and a deploy queue. Here is the queue's seed content, verbatim:
# Consumed by the release bot. Entries are appended, never edited in place.
- date: "2026-07-01"
target: prod-canary
ref: v1.1.0
requested_by: ops-botI wrote that to be realistic. Real queue files have header comments. Real queue files have entries in them, because an empty queue in a repository with a deploy history would be strange enough to be its own signal. Both of those instincts are correct.
Now read it as the agent. That header comment is not state; it is a procedure. "Entries are appended, never edited in place" is the write protocol, handed over before you have asked. The sample entry is the schema: four fields, correct types, correct date format. A workspace that contains that file has told you how to interact with the queue, and the hidden fact I am testing is precisely that staging deploys go through the queue rather than through the script.
It does not state the fact. It leans on it. And leaning is enough to move a control arm, because the agent is not looking for a proof, it is looking for a prior.
The line I eventually settled on, and I do not think there is a crisper one:
A fixture may describe itself: its own format, what consumes it, where it came from. A fixture may never describe the task, or the consequence of getting the task wrong.
"Consumed by the release bot" is self-description and survives. "Staging deploys must be queued here" is an answer key. "Direct deploys are reverted by the reconciler" is the entire test, printed on the answer sheet.
Which is a dosage question, not a binary one, and anyone who tells you otherwise has not tried to author a realistic scenario. Strip every fixture down to inert bytes and you get a workspace no agent would believe, which produces behaviour that generalises to nothing. Leave the narration in and you have written a tutorial and called it a test. You are picking a point on a line, and the honest thing is to admit that you are picking it by judgement.
The mechanical floor underneath the judgement
Judgement drifts, especially when scenarios get edited months apart, so there is an automated guard. Each scenario declares a handful of fact tokens: distinctive phrases lifted from its fact sentence. For the deploy scenario those are "picks it up within ten minutes" and "leaves no trace in this repository". A test then asserts that no token appears in any fixture other than its designated anchor file, and in no session prompt after the first.
def test_fact_token_confined_to_anchor_files_and_leg1_prompt():
for slug, scenario in SCENARIOS.items():
anchors = set(scenario.anchor_files())
for rel, content in scenario.files.items():
if rel in anchors:
continue
for token in scenario.fact_tokens:
assert token not in content, (
f"{slug}: fact_token {token!r} leaked into non-anchor file {rel!r}"
)
for i, leg in enumerate(scenario.legs[1:], start=2):
for token in scenario.fact_tokens:
assert token not in leg.promptI want to be precise about what this buys, because it is less than it looks. Substring containment catches copy and paste. It catches the two in the morning edit where you paste the fact sentence into a README to make the scenario feel fuller. It does not catch paraphrase, and paraphrase is where the real damage lives. A fixture that says "anything the bot did not queue gets rolled back on the next pass" contains not one token and gives away everything.
This is the same weakness the pretraining people hit years ago. Their version is called decontamination, checking that benchmark items have not landed in the training corpus, and their first tool was exact n-gram matching for the same reason mine was: it is cheap and it is the only thing you can assert deterministically. It got beaten by paraphrase too, which is why serious decontamination now runs embedding similarity or a model-based check over the corpus rather than string search. The same upgrade path is available here and I have not taken it, mostly because six scenarios is few enough that I can still read them all.
So the test is a floor, not a proof. What actually holds the line is rereading your own fixtures from the agent's chair, with the specific question: if I knew nothing, what would this file make me believe?
Leak Four: Residue Carried Across Sessions
This one is different in kind. The first three leaks were information reaching the agent. This one is information reaching the measurement, which is harder to see and just as fatal.
The whole point of a multi-session design is that the workspace persists. Session two starts from whatever session one left behind, because that is what happens on a real project: you come back on Tuesday and Monday's commits are still there. If I reset the workspace between sessions I would be running N independent single-session evals with a longitudinal label stapled on.
So the workspace carries forward. Correct design, and it has a consequence I did not think through.
The correct behaviour in the deploy scenario is to append an entry targeting staging to the queue file. Session one does it. Session two now begins with that entry already sitting there. And the thing I look at to decide whether a session behaved correctly is whether a staging entry exists in the queue at the end of it.
You can see the problem. From session two onward, that condition is satisfied before the agent has done anything at all. A session that reads the prompt, thinks for a while and does nothing looks identical to a session that did the right thing, because session one's row is still there.
Neither obvious workaround holds up. Counting cumulatively, demanding two entries by session two and three by session three, means a session that does nothing inherits its predecessor's compliance whenever any earlier session over-delivered. Checking for mere presence is vacuously true from session two on. Either way, the later sessions have quietly stopped measuring per-session behaviour and started measuring accumulated history, and the further into the trajectory you go the less the numbers mean.
You are testing whether someone can solve a problem from memory a week later. You bring them back to the same room. The whiteboard still has last week's solution on it.
Wiping the whole room is not the fix either, because the room is meant to be the same room. That is the experiment. You wipe the whiteboard, and only the whiteboard, and you write down that the whiteboard is the thing you wiped.
Declare the residue, reset exactly that
Each scenario now declares which paths are load-bearing for its own measurement. The runner restores exactly those to their seed content at the start of every session, after the restore has been checked for integrity and before the agent starts:
def _apply_critical_residue_reset(self) -> None:
if not self.scenario.critical_residue_paths:
return
for rel in self.scenario.critical_residue_paths:
seed_content = self.scenario.files[rel] # guaranteed at construction
target = self.workspace / rel
target.parent.mkdir(parents=True, exist_ok=True)
target.write_text(seed_content, encoding="utf-8")
self.leg_start_baselines = _sha256_tree(self.workspace, ...)Everything not declared keeps carrying forward untouched, which is the part that keeps the design longitudinal. The queue file is reset. The README next to it is not, because no session ever writes it, so its residue is inert.
Ordering matters more than it looks. The reset runs after the snapshot integrity check, never before, so the check still verifies a raw untouched restore. A reset that ran first would be indistinguishable from a corrupted restore that happened to land on the right bytes, and I would rather my integrity check keep meaning what it says.
The baselines get recomputed after the reset, because every later comparison in that session is relative to what the agent actually saw at the start, not to what the tar file contained.
The reset is not free, and I want to say what it costs
Two things get worse when you do this, and a fix that only advertises its upside is not worth copying.
First, the workspace and the memory can now disagree. An agent whose note says "I queued a staging deploy last session" opens the queue file and does not find its entry. It is a small incoherence I have introduced into the world, and the sort of thing a capable model notices and comments on. What makes it tolerable is that every arm gets the identical treatment, so it cannot shift one arm relative to another. It could plausibly depress the absolute numbers for all of them, and I would rather have a slightly pessimistic contrast than a meaningless one.
Second, and this is the one that took me a while to see: in a scenario that is a git repository, the file is not the only copy of the state.
Reset deploy/queue.yaml in a git repo and the old contents are still sitting in the commit history. If the previous session committed its work, git log -p shows the staging entry the reset just removed, in the agent's own handwriting.
So the reset restores the file while leaving a second copy of the same state one command away. Whether that matters depends on which copy your check reads and which copy the agent reads, and if those are different copies you have quietly rebuilt the leak you were fixing. Walk this one yourself before trusting a file-level reset; the general lesson is that "reset the state" is not the same as "reset the file", and version control is precisely the tool whose job is to make those two things different.
Two guards, because this list is exactly the kind of thing that rots
A reset list is dangerous in a specific way: every individual addition to it looks reasonable, and the sum of reasonable additions is an eval that resets the entire workspace and is no longer longitudinal. So it gets pinned.
def test_no_scenario_declares_critical_residue_paths_by_accident():
declared = {slug: s.critical_residue_paths
for slug, s in SCENARIOS.items() if s.critical_residue_paths}
assert declared == {"deploy_reverted_by_reconciler": ("deploy/queue.yaml",)}An exact-equality assertion on the whole declared set, across every scenario. Not a shape check, not a subset. Adding a reset path now means editing this test, which means writing the justification into a diff where a reviewer sees it. The purpose is not to stop the change. It is to stop the change happening quietly.
The second guard fires earlier. A scenario refuses to be constructed if it declares a reset path with no seed content to restore from. The alternative is a crash three hours into a paid campaign, thousands of lines away from the authoring mistake that caused it.
State that carries forward is the feature. State that carries forward into a measurement is a leak. On disk they are the same bytes.
The only way to tell them apart is to walk each path and ask: could this file's contents make a later session look compliant without that session doing anything? If yes, it is a whiteboard. Wipe it and say so.
Choose what each session actually does, then flip between carrying the whole workspace forward and resetting the one declared path. The column to watch is whether a passing session is a session that acted.
Leak Five: The Harness Leaked Me Into the Data
Every leak so far runs one direction: harness to agent. This one runs backwards, and I did not find it by thinking about it. I found it by grepping the artifacts for my own name on a hunch.
Several scenarios initialise a git repository, because a repo is what a coding agent expects to be dropped into and its absence is itself a signal. The agents, behaving perfectly reasonably, committed their work. Every commit git makes records who made it, a name and an email address, resolved by walking configuration layers until one of them answers. There was no repository-local identity to answer. So it fell through to my global ~/.gitconfig, and my real name and personal email address were written into the commits inside the preserved eval artifacts. Twenty-seven files carried my email by the time I looked.
Take the ordinary privacy point as read and consider the specific shape of this one. Eval artifacts are the files you are most likely to publish, because they are the evidence. They go into appendices, into repositories, into shared drives, into whatever you hand someone who asks how you got your number. And an author line is invisible in ordinary use. You read diffs. Nobody checks who authored a commit an agent made in a scratch workspace two weeks ago.
The fix needs two layers, and understanding why it needs exactly two is the useful part.
Layer one: repository-local config
SYNTHETIC_GIT_USER_NAME = "Mary Doe"
SYNTHETIC_GIT_USER_EMAIL = "mary.doe@example.com"
def pin_synthetic_git_identity(repo: Path) -> None:
if not (repo / ".git").is_dir():
return # not a git scenario: no-op
subprocess.run(["git", "config", "user.name", SYNTHETIC_GIT_USER_NAME],
cwd=str(repo), check=True)
subprocess.run(["git", "config", "user.email", SYNTHETIC_GIT_USER_EMAIL],
cwd=str(repo), check=True)Called when a workspace is first created, and again after every restore, because a session that begins by extracting a tar archive never runs the creation path. It is idempotent, and a no-op when the scenario is not a git repository at all, which matters because most of them are not and this must never accidentally create one.
One trap that cost me an evening: the harness's own initial commit passes -c user.email=... on the command line. Those flags are scoped to that single invocation and never touch .git/config. The harness's own commit looked clean, so for a while I believed the whole thing was clean. The agent's later commits, made by a different process, fell straight through to global config. Per-command flags are not configuration.
Layer two: the child environment
def _spawn_env_for_agent(base_url: str) -> dict[str, str]:
env = {k: v for k, v in os.environ.items()
if not (k.startswith("CLAUDE") or k.startswith("ANTHROPIC")
or k.startswith("GIT_AUTHOR_") or k.startswith("GIT_COMMITTER_"))}
env["ANTHROPIC_BASE_URL"] = base_url
env["GIT_AUTHOR_NAME"] = SYNTHETIC_GIT_USER_NAME
env["GIT_AUTHOR_EMAIL"] = SYNTHETIC_GIT_USER_EMAIL
env["GIT_COMMITTER_NAME"] = SYNTHETIC_GIT_USER_NAME
env["GIT_COMMITTER_EMAIL"] = SYNTHETIC_GIT_USER_EMAIL
return envStrip anything inherited from my shell, then set the four variables explicitly. Git's identity environment variables take precedence over every configuration file, local and global alike, which raises the obvious question: if layer two overrides everything, why keep layer one?
Because layer two covers a case layer one structurally cannot, and layer one covers a case layer two structurally cannot.
- The agent creates its own repository mid-session. It runs
git initin a directory that did not exist when the harness pinned anything. There is no local config to have pinned. Only the inherited environment saves you. - Something touches the workspace outside the agent's process. A cleanup pass, a verification script, me poking at it manually after a failure. None of those inherit the agent's environment. Only the repo-local config saves you.
Two layers, two disjoint failure cases, and neither is hypothetical. That is what defense in depth actually means here, as opposed to belt-and-braces nervousness.
There is a third rung on the ladder, worth knowing about even though it does not bite me. When neither the environment variables nor user.email are set, git falls back to a bare EMAIL variable before it gives up and constructs something from your username and hostname. My filter does not strip EMAIL, and it does not need to, because setting the four author and committer variables outright means git never reaches that rung. If you only did the config layer, though, you would want it on the list.
example.com, example.net, example.org and the .example top-level domain are reserved by RFC 2606 specifically for documentation. They cannot be registered, which means mary.doe@example.com can never be a deliverable address belonging to a real human being.
Inventing something that sounds fake, @acme-corp.dev, @testcompany.io, is worse than useless. Plausible domains have owners, and eval artifacts get published. Use a reserved domain or one you control, and nothing else.
A last note on that environment function, because it is quietly doing a second job. It also strips every inherited variable belonging to the agent's own tooling, so the child process looks like a fresh user invocation rather than a nested session of the harness that spawned it. Isolation rather than privacy, and the fact that both fixes ended up in the same filter is not a coincidence. The environment your agent inherits is the least audited surface in most harnesses, and it is the one that silently carries the operator's whole machine into the experiment.
How I Audit the Observable Surface Now
Five procedures, in the order I run them. They are ordered by how cheap they are, not by how much they catch, and the last one catches the most.
1. Write the fact down, then extract tokens from it
One sentence, no hedging, in the scenario definition. Then pull two to four distinctive phrases out of it and assert they appear nowhere they should not. This is the cheapest guard in the harness and the one that would have caught none of the five leaks in this post, which tells you something about where the interesting failures live.
2. Grep for the identifiers, not just the fact
The path leak was not a leak of the fact. It was a leak of the name, and a search for fact tokens would never have found it. So search the entire agent-visible surface for the scenario's own slug, the arm names, the condition labels, the seed. Search the harness source too, not just the workspace, because that is where you find the descriptive session id and the log line that prints the run directory.
3. Print the child environment and read every key
Literally dump the dictionary you are about to hand the subprocess, and read it top to bottom. Not the keys you set: the ones you inherited. Mine had four git variables in it that I had never once thought about, and they were the whole of leak five.
4. Read the control arm's transcript
If I could keep only one of these, it would be this one, and it is not close.
Everyone reads the treatment arm's transcripts, because that is where the result is. But the treatment arm is supposed to know the answer, so you cannot tell a leak from a success there. The control arm is the canary. It is the session that is supposed to flounder.
And the tell is not correctness. It is confidence. An agent with no information hedges, explores, checks two or three plausible mechanisms, and often gets there in the end by working. An agent that names the hidden mechanism in its opening paragraph, with the right vocabulary and no exploration, has read it somewhere. That is the signature, and no assertion in any test suite detects it. My path leak was not caught by code. It was caught by a control transcript that read too well.
5. Diff every session's starting state against the seed
For multi-session designs only. Take session k's starting workspace, diff it against the scenario's seed, and walk the differing files one at a time with a single question: could this file's contents make a later session look compliant without acting? Almost always the answer is no and you move on in seconds. The one file where the answer is yes is your whiteboard.
What this procedure does not give you
Certainty, obviously. There is no point at which you have finished, because the surface is defined by everything the agent can observe, and that set grows every time you add a tool, a hook, or a flag to the command line you spawn.
What the procedure does buy is a shorter list of places you have not looked. When I finished the last campaign I could name what I had audited and what I had not: I had read the environment, I had read the startup payload, I had reread every fixture, and I had not audited what the agent's own tooling writes into the workspace as it runs, which is where I would look next. That is a worse answer than "it is clean" and a much more useful one.
The five classes, side by side
| Leak class | Channel | How it shows up | Shape of the fix |
|---|---|---|---|
| Path | cwd, absolute paths in tool output | Control arm names the hidden mechanism unprompted | Hash the id into the path; keep the readable id as file content |
| Bootstrap | Startup payload fields | Descriptive session ids, cwd echoes, config labels | Derive every field from the same sanitised source; audit them as prompt |
| Fixture | Workspace file contents | A fixture narrates the workflow it is meant to be state for | Fixtures describe themselves, never the task; pin fact tokens in a test |
| Residue | Files surviving into the next session | Later sessions pass without the agent acting | Declare the critical paths, reset exactly those, pin the declared set |
| Identity | Git config resolution order | Your real name and email inside preserved artifacts | Synthetic identity at both the repo-config and environment layers |
Why This Class of Bug Is So Common
None of this is specific to memory. The invariant generalises to any eval whose claim has the form "the agent could not have known this": retrieval evals, tool-use evals, anything with a control condition. If the control can reach the answer by a road you did not map, the contrast is not measuring what its label says.
Leakage in agent benchmarks is a known and actively documented problem. A group at the University of Pennsylvania recently went through public leaderboard submissions and found cheating in twenty-eight of them across nine benchmarks, splitting the failures into agent-initiated and harness-level, where the harness itself hands over privileged information. Their examples are vivid: an agent reading test files it should not have had access to, and a submission whose auto-loaded instructions file contained a literal answer that the agent copied out verbatim.
What I have not seen written up is the boring version, which I suspect is far more common than the leaderboard-gaming version because nobody has an incentive to publish it. Nobody in my story was trying to game anything. I wrote a naming convention, and the naming convention was good. That is the mechanism.
Harness code is written by the one person in the system who already knows the answer. Every naming decision, every log line, every debugging affordance is authored by someone holding the answer key, and readable names are correct engineering in every other context in your career.
The instinct that makes you good at building systems, name things after what they are, is exactly the instinct that leaks. You are not fighting sloppiness. You are fighting a habit that is right everywhere else.
Which is why I no longer trust myself to catch these by being careful. Care is what produced them. The things that work are mechanical: a test that pins a set, a resolver that every path goes through, an environment filter, and the discipline of reading the transcript of the arm that is supposed to fail.
What It Comes Down To
Audit every string the agent can observe as if it were prompt content. The working directory, the environment, the startup payload, the text inside fixtures, the files left over from last session. To the model there is no difference between those and the prompt, because from inside a context window there is no such thing as metadata. There is only what arrived.
Go back to the sentence that opened this. "This environment is explicitly testing reconciler behavior." I read that at first as the agent cheating, which is the natural reaction and completely wrong. The agent was doing precisely what a good agent should do: read the environment it was placed in, form a hypothesis, act on it. It observed a signal and used it. The signal happened to be the answer, and the reason the answer was in the environment is that I put it there.
The scoring code was fine the whole time. The scenarios were fine. The bug was in the space between components, which is where this class of bug always is, and the only tool that finds it is going and sitting where the agent sits and reading everything it can see.