Relevance Is the Wrong Sort Order for Agent Memory

Ranked retrieval by probability of relevance is one of the load-bearing ideas in information retrieval, and it is a good one. The Probability Ranking Principle (PRP) is the statement, and under two stated conditions it licenses exactly the kind of sort every retrieval system does today. S. E. Robertson stated the principle in 1977; the standard references are his 1977 paper and the longer treatment in van Rijsbergen's Information Retrieval (1979).

Two of the principle's conditions matter here.

The first is that document relevance is independent. The standard statement of the principle, as given in the Wikipedia summary of the original derivation and reproduced in surveys, assumes the relevance of one document is independent of the relevance of any other. That is what licenses a simple sort: if the value of each item is intrinsic and unrelated to the others, ordering by that value is optimal and you are done.

The second is that the user integrates. Search returns a list to a person who reads several results, notices that two disagree, and resolves the conflict themselves. The system is not obliged to deliver a coherent answer, only a well-ordered set of candidates the user can arbitrate.

Information retrieval knows the independence assumption is not always safe. The whole point of the diversity and redundancy literature is that item interaction is real, and a ranked list of near-duplicates is worse than its individual scores suggest. Maximal Marginal Relevance (MMR), introduced by Carbonell and Goldstein at SIGIR 1998, makes the trade-off explicit: a linear combination of query-relevance and novelty against the items already selected. The MMR formula, in the notation of the original paper, is

MMR criterion (Carbonell and Goldstein, SIGIR 1998)
MMR = argmax_{D_i in R \ S} [ lambda * Sim_1(D_i, Q) - (1 - lambda) * max_{D_j in S} Sim_2(D_i, D_j) ]

Read that: keep some lambda weight on similarity, subtract a penalty proportional to the maximum similarity to anything already chosen. The penalty is exactly the operation that improves a redundant list, and it is the operation that hurts a list of corrections.

But redundancy is a mild violation. Two documents saying the same thing waste a slot. Memory has a severe violation: two notes saying opposite things, where one of them is the retraction of the other. No amount of diversity tuning addresses that, because the problem is not that the items overlap. The problem is that one of them determines whether the other is still true.

The PRP is correct in the regime it was derived for. Memory retrieval is not in that regime, and the derivation has not been re-done for the regime it is being applied to. Every agent memory system that ranks by similarity is running the PRP with the wrong problem statement, and getting the wrong answer in a specific, structural way that the rest of this post is going to make concrete.

A note in an agent memory system is not a document. It is a move in a sequence, and it is frequently defined against what came before it.

"Use the venv interpreter for the test suite" is a note. "That was because the global interpreter lacked a grammar package, which was fixed in the July upgrade" is a note about that note. "The venv rule still applies because CI derives from it" is a note about that one. Each is a coherent statement on its own. Delivered in isolation, the first is a rule with an unknown reason, the second reads as a repeal, and the third is meaningless.

Every memory system with a lifecycle has the edges to express this. Supersession, revocation, correction, and refinement all exist as first-class relations in the better designs, including in the arguments I have made previously about expiry as a state and about reinstatement as the missing transition. That post argued revocation needs an inverse, so a note proven wrong can come back when it turns out to have been right. This post is the layer above: once notes carry supersession and revocation edges at all, ranking them independently by similarity is the wrong retrieval contract, because the correction sits closest in embedding space to the thing it corrects and gets dropped by the very machinery meant to reduce redundancy.

What has not happened is those edges being made load-bearing at retrieval time. They are stored, they are queryable, and then the recall path ranks by cosine similarity and returns a set, and the edges contribute nothing to what the caller receives.

The gap is not in the data model. It is in the delivery.

A document is about a topic; it is judged by its own content against the query, and the relevance of two documents is plausibly independent because each makes its own claim about an external subject. A note is about another note: the supersession edge is part of the claim. Returning one member of a supersession pair is, in retrieval terms, returning a sentence that has been formally declared false, with a high confidence score, and asking the caller to integrate it. The caller will not know to integrate it, because the integration is precisely the lifecycle edge that the ranker was designed to ignore.

This is the part that turns an abstract objection into a concrete failure, and it is the strongest argument in the post.

Ask which pairs of notes in a store are nearest each other in embedding space. The answer is: notes about the same subject, in the same vocabulary, at similar length. Now ask which pairs of notes stand in a supersession or revocation relation. The answer is the same: a correction is about the thing it corrects, and it reuses its vocabulary, because that is what a correction is.

The relation the ranker most needs to respect is concentrated exactly in the region the ranker treats as redundancy.

Two consequences follow, and both are bad.

The first is a plain retrieval effect. Under a top-k cutoff, near-duplicate notes compete for adjacent slots, so a superseding note and its predecessor are frequently separated by the boundary rather than both included. Which one lands inside depends on lexical accident: which one happens to share more surface form with the query. A correction phrased tersely ("this was wrong, the cause was the proxy") often shares less surface form with a topical query than the confident, well-written original it corrects, so the ranker prefers the note that has been retracted.

The second is worse, because it comes from the system's own quality machinery. Any diversity or deduplication pass, added in good faith to stop the store returning five paraphrases of one thing, will look at a note and its correction and see a near-duplicate pair, and will drop one. The one it drops is typically the one that adds the least new lexical content, which is, again, the correction. The mechanism installed to improve result quality systematically removes the notes that carry the truth.

The interactive demo below makes this visible. It seeds a small store of notes that pair every claim with its correction, then runs two retrieval regimes against a topical query. Regime A is plain top-k by similarity. Regime B closes the result on the supersession edge: a note and its successor are always returned together, regardless of independent similarity scores. Watch what Regime A keeps and what it loses.

The obvious counter is to sort by time instead. It fails, and understanding why is what points at the actual answer.

Recency is not causality. A note written after another is not thereby about it, does not thereby correct it, and does not thereby supersede it. A store sorted by recency delivers whatever was written most recently on any subject, which for a topical query is frequently an unrelated note from a busy afternoon.

And the inverse error is worse. An old note is not thereby superseded. The most durable knowledge in a codebase is often the oldest: why a migration was written a particular way, which constraint a design was working around. Sorting by time promotes the recent and demotes the durable, and time is measuring elapsed duration, not validity. Age may inform ranking. It must never decide validity, and a recency sort quietly lets it.

The position is stated at length in a previous post on expiry as a state: that post argued that age is not evidence about whether a note is still true, and the present post takes the same observation as given and extends it. The argument here is that recency-as-sort is not just a weak proxy, it is the wrong axis in the same way similarity is the wrong axis, for the same kind of reason: it confuses a surface feature with a causal one.

The useful observation buried in the recency proposal is that it is reaching for order, and specifically for an order in which a correction lands after the thing it corrects. But global time order is a poor proxy for that. What is actually wanted is the edge: this note supersedes that one. That edge exists. It is already stored. It is simply not being used at the moment the results are assembled.

The empirical shape of this point comes from a different source. LongMemEval (Wu et al., ICLR 2025) tests long-term interactive memory with five abilities including knowledge updates, where a fact stated early becomes outdated later, across 500 questions embedded in sustained chat. The paper reports that commercial chat assistants and long-context LLMs show a roughly 30% accuracy drop on remembering information across long interactions, and identifies knowledge updates as one of the harder abilities in the suite. The exact mechanism that drives the drop is not isolated in the paper, and the store in LongMemEval is the chat history rather than a curated memory. The relevant lesson for this post is that a retrieval system whose primary axis is unrelated to the fact that an update happened will struggle on knowledge updates, and the ranker's knobs (similarity, recency, diversity) are not the axis that would help.

Here is the alternative, stated as a design.

The unit of delivery is not a note. It is a belief and its trajectory: the current state of a claim, plus the events that produced it, plus the notes it superseded or that revoked it. Relevance still runs, and it still does the thing it is genuinely good at, which is choosing where to enter the graph. What it no longer does is decide what comes back.

Two rules define the behavior.

Closure. If a note is returned, every note that supersedes or revokes it is returned with it, whether or not that successor is independently similar to the query. Its relevance is inherited: the successor is relevant because the predecessor was, and the caller cannot correctly use the predecessor without it. This is the rule that makes the failure in Part 3 structurally impossible rather than merely less likely, and it is deliberately unconditional. A successor that scores badly on similarity is exactly the case the rule exists for.

Terminal state first. Within a returned chain, the current state leads and the history follows, marked as history. The caller should not have to reconstruct which note won. That reconstruction is the system's job, and it is cheap for the system and error-prone for the caller.

The result is not a ranked list of notes. It is a small number of chains, ordered by the relevance of their entry points, each internally ordered by causation rather than by score.

Note what this is not. It is not a graph traversal that pulls in everything topically connected, which would blow the budget and reintroduce noise. The closure is over lifecycle edges only, which are sparse, deterministic, and recorded at write time rather than inferred. Most notes have none, so most chains are one note long and the output looks exactly like today's. The difference shows up only where it matters.

To state the link to the earlier post: that one argued that revocation needs a reverse edge, reinstatement, so a note can come back when the revocation turns out to have been wrong. This post argues the parallel point for retrieval. The store has the edge. The store does not use it. Closure makes the edge load-bearing at the moment the caller is about to be misled by its absence.

Four consequences follow, and an honest argument states them rather than leaving them for the reader to find.

A note can no longer be scored in isolation. Scoring is per chain, and a chain's value is not the sum or the max of its members. This complicates every ranking heuristic that assumes a flat candidate pool, and it means the retrieval and the assembly stages can no longer be cleanly separated. The retrieval stage chooses entry points, the assembly stage composes chains; the two cannot be designed independently.

The top-k contract breaks. Ten notes might be four chains, and the caller asked for ten of something. Either the contract becomes "k chains" with variable note count, or "k notes" with the closure rule sometimes forcing an overrun. There is no formulation that keeps both, and I think the honest resolution is that the contract was always slightly wrong: what a caller wants is a budget, in tokens, not a count of items. A chain is a natural packing unit for that budget; a sorted list of independent notes is not.

Deduplication becomes dangerous by default. Any near-duplicate suppression has to become lifecycle-aware, or it reintroduces exactly the Part 3 failure. In practice this means dedup must never be allowed to drop a note that stands in a lifecycle relation to another candidate, which is a constraint the usual implementations do not have a place to express. The MMR reranking in the demo is a model of the issue: it is doing the right thing by its own definition and the wrong thing for this contract.

Budgeting gets harder. A chain has variable cost, so filling a fixed token budget with whole chains is a packing problem rather than a prefix of a sorted list. It is a small packing problem, and the practical answer is probably to take chains greedily by entry-point relevance and stop, but it is no longer trivial and it should not be pretended otherwise. A ranker that pretends the budget is just a count of items is the ranker that delivered the Part 3 failure in the first place.

None of these is fatal. All of them are the price of delivering results that are true rather than results that are well ordered.

No measurement is offered. The argument is structural: it identifies an assumption imported from document retrieval, shows a specific mechanism by which it fails for notes, argues that the obvious alternative axis fails too, and derives a delivery rule. That is an argument, not a result.

The experiment is specifiable and cheap, which makes the absence of a measurement here a choice rather than an excuse. Build a store containing correction chains, where a note is followed by a note that revokes or supersedes it. Issue topical queries. Measure how often relevance-ranked top-k returns a superseded note without its successor, and how often a diversity pass drops the successor specifically. Then measure the same with the closure rule enabled, and measure what it costs in budget. The first number is the one that decides whether this matters: if superseded notes almost never separate from their successors in practice, the argument is correct in principle and unimportant in practice, and that is a finding worth publishing too.

This does not claim similarity search is the wrong tool. It is the right tool for the job it should be doing, which is finding the entry point. The claim is narrower and sharper: it should not be deciding the composition of the result. A retrieval that uses similarity to choose a chain head and lifecycle edges to assemble the chain uses each tool for the thing it is good at. A retrieval that uses similarity to choose every note uses it for the thing it is bad at, with consequences the rest of this post has tried to make concrete.

None of this addresses conflicting notes with no recorded edge between them. If two notes disagree and nobody ever recorded that one supersedes the other, no delivery rule can fix it, because the information does not exist. That is a capture problem and a genuinely open one, and the closure rule is worth nothing on a store where the edges were never written. The store has to be written with the lifecycle in mind, the way the store in the reinstatement post is written with the reverse edge in mind. The store can do the right thing for the data it has, and the data has to be there.

• Robertson, S. E. "The Probability Ranking Principle in IR." Journal of Documentation 33(4), 1977. Source of the probability ranking principle. The relevance-independence assumption is the one Part 1 leans on. Wikipedia summary reproduces the assumption in standard form.
• Carbonell, J., Goldstein, J. "The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries." SIGIR 1998. MMR formula and the relevance-novelty trade-off. Original paper. Part 1 and Part 3.
• Saha, S. Agent Memory Expiry Is a State, Not a Delete. The prior post on age and validity. Cited from Part 4 rather than re-derived. Argues that elapsed time is not evidence about a note's truth.
• Saha, S. Reinstatement Is the Missing Primitive in Agent Memory. The prior post on the reverse edge of revocation. Part 2 and Part 5 of this post argue the parallel point for retrieval: the store has the edge, the ranker does not use it. arXiv:2607.20972 is the underlying research paper.
• Wu, D., Wang, H., Yu, W., Zhang, Y., Chang, K.-W., Yu, D. LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory. ICLR 2025, arXiv:2410.10813. Five abilities including knowledge updates, 500 questions, reported 30% accuracy drop across long interactions. Part 4 cites it for the empirical shape of the recency problem, not the mechanism.
• Chhikara, P., Khant, D., Aryan, S., Singh, T., Yadav, D. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. arXiv:2504.19413, April 2025. ADD/UPDATE/DELETE/NOOP update phase. Part 3 and Part 5 reference it as a production system that stores the supersession edge but does not close retrieval on it.
• Wang, Z. TOKI: A Bitemporal Operator Algebra for Contradiction Resolution in LLM-Agent Persistent Memory. arXiv:2606.06240, June 2026. Audit rows, isolation preconditions, default-retrieval filtering. Part 3 and Part 5 reference it as the production system with the cleanest auditability story, and the same gap between auditability and successor-closure at default retrieval.
• vectr source: agent/working_context_store/_events.py (NOTE_EVENT_KINDS) and _store.py (note lifecycle fold). The supersession and revocation event vocabulary over notes, documented publicly in the prior two posts in this series.