Building Vectr — Part 2 of 3

Most people treat /compact as "clear the context to keep going." That framing is roughly correct but understates the damage. The issue isn't just that context gets shorter — it's that the compression is lossy in exactly the cases where being wrong is most expensive.

/compact works by asking the AI to summarize the current conversation, then replacing the full history with that summary. Token count drops from (say) 180,000 to 12,000. You continue from the summary. Here's what the summary doesn't preserve:

Exact function signatures

A summary might say "the function takes a path and a flag." The conversation had def process_workspace_changes(path: Path, db: Database, *, force: bool = False) -> list[ChangeResult]. The difference between those two descriptions is the difference between a valid call site and a runtime error.

Specific line numbers

"The resolver module" and /src/workspace/resolver.rs:214 are not the same precision. You can reconstruct the file path, but it costs you a tool call — and often another read of the surrounding code to re-establish context.

Non-obvious behavioral invariants

If you spent three turns establishing that acquire_lock() must be called before touching workspace metadata because there's a race condition with the filesystem watcher, that three-turn understanding might survive as "be careful with locking." The exact invariant — the one that matters when you're writing the code — is gone.

The reasoning chain

Sometimes the value of an exploration session isn't the final answer but the chain of observations that produced it. "We looked at X, noticed Y, which suggested Z" — this chain is what you need if the conclusion turns out to be wrong later. Summaries discard chains. They keep endpoints.

In the CPython scenario, re-establishing the finalizer ordering invariant from scratch means re-reading several files and re-following a non-obvious call chain — roughly 15–20 minutes of work that was already done. A note stored at the end of session two takes about a minute to write and ten milliseconds to retrieve. Session three starts with vectr_recall("garbage collector finalizer ordering") and picks up from actual precision rather than reconstructing from a fragment.

When I started building Vectr's memory layer, I had a clean model: the AI finds something useful, stores it with vectr_remember, then drops the file from its context window. The note is 50 tokens. The file was 800 tokens. Net gain: 750 tokens freed for new content. I called this "context offload."

I built it this way. I wrote documentation describing it this way. I designed vectr_evict_hint entirely around it — a tool that identifies which retrieved chunks the AI had already "processed" and could therefore "drop."

It doesn't work. And the reason explains something important about how transformer models handle context.

The KV cache is append-only

Think of the transformer's memory as a lookup table it builds as it reads each token. For each token it processes, it computes a key-value representation that gets stored at each attention layer. Every subsequent token the model generates attends back to every previous token through these cached representations — that's how earlier context influences later output.

Once a token's representation is computed and cached, it stays until the context is cleared. There is no mechanism to evict specific tokens by instruction. "You can drop chunk X from your context window" is itself processed as tokens — added to the cache, not used to remove other entries from it.

A subtlety worth naming: the KV cache as described here is the model's internal representation, maintained server-side by the inference provider. What you see as "context window usage" is a count of tokens in the current conversation, not a direct readout of GPU memory. The principle holds regardless: every token in the conversation occupies a slot in the cache, and you cannot remove individual tokens from a running session without ending or compressing the whole thing.

I measured context window usage before and after sequences of vectr_remember + vectr_evict_hint calls: essentially unchanged. The hint was adding tokens to the cache while accomplishing nothing at the context management level. In some cases it made things marginally worse.

This misconception cost real development time. I built vectr_evict_hint to fire automatically when cumulative retrieved token count crossed a threshold, expecting the AI would see the hint, store notes, and drop files. What actually happened: the AI would see the hint, store notes (genuinely useful!), and then also keep the files in context (because nothing dropped them). Benchmarks with this framing baked in gave confusing numbers — sometimes Vectr cost more than the baseline, even on tasks where it should help. The accumulated search results staying in context without any actual eviction was part of that.

Once I dropped the context-offload framing, the actual value of vectr_remember became clear. It operates on three time horizons. The value increases as you move further in time — and the first tier is honestly the least important one.

Tier 2 is the one that makes a concrete difference in practice. When /compact fires mid-investigation, there are two scenarios. Without notes: the session resumes from a lossy summary, and you spend time re-establishing the precise understanding you had before. With notes: vectr_status() shows notes exist, vectr_recall("finalizer ordering") retrieves the exact invariant you stored, and the session continues from where the understanding actually was — not where the summary says it was.

Tier 3 compounds in a way that's easy to underestimate. The first session on a complex codebase pays the discovery cost. The second session pays it again if there are no notes, or skips it almost entirely if there are. By the tenth session, a well-maintained note store is a persistent model of the codebase that makes every session faster. The initial investment in careful note-taking earns compound returns.

The design of the notes themselves matters as much as the fact that they exist. Two principles drive everything else.

Don't store file pointers

"See resolver.rs:214 for the lock implementation" is a bad note. Why: it gives you a file path and line number. File paths change during refactoring. Line numbers change with every edit. A note from three days ago that says "line 214" will be wrong by the time you use it if anyone has touched the file.

More fundamentally: a pointer hasn't captured what you learned. It's a reference. When you recall it, you still have to read the file to extract the actual knowledge. You've stored the address, not the content at the address.

Store the finding itself

A good note, using the same codebase as an example:

This note is about 120 tokens. Reading the relevant files to reconstruct this knowledge would cost 600+ tokens plus two turns. The note captures the actual insight — the non-obvious invariant about lock order — not just a pointer to where to find it. When vectr_recall surfaces this in a later session, the AI continues from the right understanding immediately.

Priority and tags are not cosmetic

vectr_remember takes priority ("high", "medium", "low") and tags (a list of strings). Priority affects recall ordering: high-priority notes rank higher when multiple notes match a query with similar relevance scores. In a long investigation where many findings are equally relevant, high-priority notes surface first — the ones you explicitly flagged as critical.

Tags enable filtered recall. vectr_recall(query="locking", tags=["concurrency"]) returns only notes tagged with "concurrency" that semantically match the query. On a codebase with hundreds of stored notes accumulated over months of work, filtering by subsystem tag makes recall precise without returning results from unrelated parts of the codebase.

For several early benchmark runs, vectr_recall was firing in implementation sessions but returning nothing useful. The benchmark showed 0 relevant recall results across 5 separate implementation sessions on CPython tasks — even though the research session had stored detailed notes about exactly the functions being modified.

The root cause took some logging to surface: recall was using SQL LIKE queries against a text field, not semantic search against the vector store.

SQL LIKE is substring matching. vectr_recall("garbage collector finalizer ordering") would only return notes that contained the exact string "garbage collector finalizer ordering" somewhere in their text. A note about PyObject_GC_Del describing finalizer behavior — stored with different wording in a different session — wouldn't match unless it happened to contain those exact words in that exact order.

The fix: use the ChromaDB vector store for recall, not the SQL table. Notes are embedded when stored and retrieved by semantic similarity when recalled.

The impact was immediate and large. In the full CPython re-run after B9 was fixed, vectr_recall fired in 4 of 6 implementation sessions with relevant results, compared to 0 of 6 in earlier runs. Per-task re-discovery costs dropped correspondingly.

This bug sat undetected for several benchmark runs because the initial benchmark design didn't make empty recalls visible. It was only when I added per-tool logging — "vectr_recall called N times, N empty responses" — that the pattern became obvious. The right signal for a recall system is not just "how many times was it called" but "how many times did it return something useful."

After fixing the context-offload misconception, I kept vectr_evict_hint in the tool set but reframed it completely. It now does something genuinely useful — just not what I originally thought.

What it actually does: it tracks the cumulative token cost of all code chunks Vectr has retrieved in the current session. When this cost crosses a threshold (default: 40,000 tokens, or more than 20 tool calls), it appends a hint to the next search response:

The important word is "re-retrievable," not "droppable." The hint doesn't claim to free tokens. It tells the AI: these files are in the index, you can get them back in under 50ms if you need them, so don't re-read them out of caution when you already have what you need in a note or could re-search instantly.

This is a behavioral nudge, not a memory management operation. The value is discouraging the pattern where an AI re-reads files "just to make sure" — burning tokens on a re-read that would have yielded the same information as a fast re-retrieval.

The threshold values — 40K retrieved tokens or 20 tool calls, whichever comes first — come from research on MemGPT (arXiv:2310.08560), which found that models begin exhibiting "lost in the middle" degradation at roughly 70% context fill. Using a disjunction (first threshold reached triggers the hint) keeps the hint from firing too late on sessions that accumulate few large files but many small searches. The 40K token count is approximate — Vectr tracks retrieved chunk tokens but not system prompt or conversation history tokens, so 40K retrieved puts real fill somewhere between 60–80%, which is where you want the prompt to land.

Knowing that notes are valuable doesn't make the AI store them. In early sessions, vectr_remember call rates were low — not because the AI couldn't see the tool, but because there was no clear trigger for "now is the moment to save this."

Saving notes is a habit humans develop from experiencing loss. An AI editor in session 1 of a new codebase has never lost anything to /compact here — it's optimizing for the task in front of it, not for a compression event that might happen three hours from now.

The pair pattern

The solution was making the save-moment explicit and concrete in the CLAUDE.md template that Vectr writes into a workspace. The original documentation said things like "save findings when they seem important." The revised version:

"Treat every search call as a pair: search, then save." This turned out to be the most effective framing. Not "save when it feels important" (too vague, relies on judgment in the moment) but "pair every retrieval with a note" (concrete, actionable, a habit that fires on a specific trigger).

The benchmark sessions that stored the most notes also had the lowest re-discovery costs in subsequent tasks. The correlation is strong enough that note count at end of research phase is a reasonable proxy for how useful the subsequent implementation sessions will be.

When not to search: the SR-RAG finding

The pair pattern addresses "when to save." There's a complementary question that ended up in the same CLAUDE.md template: when to search at all. Before calling vectr_search on a well-known API or framework, the AI should first write out what it already knows — function signatures, key types, parameter names — and only search if genuine gaps remain after that.

This comes from a paper on SR-RAG (arXiv:2504.01018). The finding: models often retrieve information they already know from training — information baked in during training, before you even opened this codebase — adding token cost without improving answer quality. Writing out what you already know before searching reduces unnecessary calls by 26–40% on familiar codebases. On an unfamiliar codebase, the AI's training knowledge rarely applies — every search turns up something new. On well-known frameworks like SQLAlchemy or React, training knowledge is often more accurate than indexed documentation. The verbalization step surfaces which situation you're actually in.

Beyond individual notes, there's a use case for checkpointing entire session states. You've spent three sessions mapping a complex subsystem, accumulated 15 detailed notes. You want a named marker that represents "the state of my understanding of the locking subsystem as of 2026-06-08" — distinct from the ongoing notes you'll add in future sessions.

vectr_snapshot("lock-subsystem-mapped") seals the current note set under a named label with a timestamp. It doesn't delete notes — it timestamps and labels the current state. vectr_snapshot_list() at session start shows all checkpoints.

The typical workflow for a complex multi-session investigation:

The snapshot name becomes the identifier for "what did I know when I made this decision." If a refactor breaks something two months later and you need to reconstruct the reasoning, the snapshot timestamps tell you which notes predated the change and which were made after it. This is the kind of temporal context that summaries and logs don't give you.

Notes can be wrong. A note about a function's behavior written before a refactor may describe the old behavior. A note about an API endpoint may describe the pre-migration version. Stale notes are worse than no notes — they give the AI false confidence in outdated information.

vectr_forget takes a note ID and deletes it from the store. Every vectr_recall response includes note IDs so you can act on them inline:

If the locking implementation changed in a refactor since May 14: vectr_forget("abc123"), then store a fresh note. The workflow is: recall → verify against current code → forget the stale note → store the updated one.

Vectr also appends a [STALE] marker automatically when a file path extracted from a note's content no longer exists in the workspace. The extraction is a simple regex scan for paths that look like foo/bar.py or src/resolver.rs — not deep parsing. When those paths disappear from the file tree, the note gets flagged. This fires on file deletion and rename, both common refactoring outcomes. It's a prompt to verify, not an automatic deletion, and it only catches path-level staleness — not behavioral changes in files that kept their names.

Looking back at the original Vectr documentation for working memory, almost every sentence led with the wrong framing. "Store to vectr, then drop from context." "Offload findings to free context budget." "Context offload layer." Every one of these is technically false, and I shipped all of them.

The correct version is shorter: store findings now so you can recall them precisely later. Through /compact. Through a new session. Through however many turns separate the discovery from the moment you need to use it. The value is in the later. The storing is cheap — a minute of attention and a tool call. The recalling is where you get the hours back.

Everything else in the system — priority, tags, snapshots, the evict_hint threshold — is infrastructure for making that recall precise. None of it matters if the fundamental framing is wrong. And mine was wrong for longer than I'd like to admit.

If I were writing the documentation from scratch I'd lead with the /compact scenario. Not with "context offload" or "memory management" — with the specific moment when a detailed understanding of a complex system compresses into a three-sentence summary that can't be acted on. That's the moment where a stored note is worth exactly what it cost to write it.