The Bugs Only Dogfooding Finds

Some vocabulary first, because the bugs are unintelligible without it, and none of it is hard.

A daemon is a program that runs in the background and answers requests. Vectr's daemon holds an index of my code and a store of notes, and it answers two kinds of caller. One is the CLI, the commands I type in a terminal. The other is my editor's AI agent, which talks to the daemon over MCP, a protocol for exposing tools to an AI assistant. Both callers reach identical logic underneath through separate thin translation layers, one per protocol, and those layers are separate code with separate bugs. That will matter in episode four.

The instance I actually live in serves nine project folders at once and runs in memory-only mode: notes and recall, no code index, no file watching. That mode exists because indexing is expensive and memory is not, a distinction episode six pays for in full. As I write this the instance holds 606 notes and has been up long enough that I no longer remember the exact command that started it. That last detail is the entire first bug.

Living inside a tool is different from testing it in three specific ways, and each one shows up below:

• The state is old. A test starts from nothing. My daemon started weeks ago with flags I have since forgotten, on a machine that has been asleep, woken, filled, and cleaned since.
• I am busy. Nobody dogfooding is dogfooding. They are doing something else, and the tool is in the way of it. That is when you follow instructions literally instead of thinking them through, which is exactly how a product's own advice gets to hurt you.
• Nothing is staged. No fixture built the workspace, no fixture chose the port, no fixture decided where on disk the checkout sits. The environment is whatever accumulated.

Five of the six bugs below are vectr's. One is arguably mine, about my laptop rather than my code, and I have included it because the boundary it exposes is the same shape as the others and because pretending your hardware is not part of your system is how you spend three days blaming an API.

Here is the mechanical reason these bugs survived a suite that catches plenty of others.

Every test builds its own world before it asserts anything. A temporary directory with four files in it. A fake service object standing in for the real one. A port the harness picked. A clock that does not move unless the test moves it. This construction is not sloppiness, it is the whole point: a test is repeatable precisely because it controls everything the code touches.

The control is also the ceiling. A defect whose cause lives outside the constructed world cannot be caught by a test that does not construct it, and constructing it means having thought of it first.

I want to be careful here, because there is a cheap version of this claim that is wrong. Nothing stops you from writing a test that starts a daemon with one set of flags, shells out to restart, and asserts the mode survived. I wrote roughly that test after the fact, and it now guards the fix. So the barrier is not expressive power. The barrier is that the test encodes a hypothesis, and every one of the six below was invisible precisely because I did not hold the hypothesis. Residency does not give you tests you could not have written. It tells you which ones were worth writing.

So the useful question is not "did we test enough" but "what is on the other side of the wall the tests are built inside." For the six that follow, the answer is a specific boundary each time: process lifetime, packaging metadata, live client connections, transport adapters, the filesystem path the code happens to sit at, and the power manager. I will come back and lay them out side by side in section nine, once the stories have earned it.

I ran a routine command and the tool printed a warning at me:

vectr: daemon on port 8765 is running older code
(1.7.0+<sha> vs 1.6.0+<sha>) - run 'vectr restart <workspace>'

Set aside that the warning was wrong, which is the next section. I did what it said. I ran vectr restart <workspace>, exactly as printed, because that is what you do with a suggested command when you are in the middle of something else.

The daemon came back in full mode. It had been running memory-only for weeks. restart reads its mode from the flags you typed on that line and nowhere else, so a restart that omits --memory-only is a restart into the default, which is full indexing. Full mode on a nine-root workspace means walking every folder, cutting every source file into chunks, and computing an embedding for every one of them. On this laptop that is not a background task, it is an event. Section eight has the numbers.

I noticed within a couple of minutes because the machine got loud. There was measurable collateral too: the persisted chunk store went from 12,092 chunks down to 9,069 across that unintended full-mode interlude. My best guess is that the re-index rewrote the store under a different exclusion state than the one that had originally built it, but I have not chased that to ground, and I would rather leave it as an open loose end than dress a guess up as a finding. The notes were untouched, all 604 of them at the time, which is the only reason this was an annoyance rather than a disaster.

The interesting part is not that restart drops the mode. It is the shape of the failure: the product's own remediation advice was the mechanism of harm. Two independently defensible decisions composed into something neither of them was. The staleness banner suggests a restart. Restart takes its mode from what you typed. Nobody wrote a bad line of code, and the bug lives in the gap between two files.

What makes it slightly embarrassing is that the CLI already knows the answer. There is a helper in it that asks a running daemon for its current mode over the status endpoint, used elsewhere for exactly this kind of question. Restart could call it before killing anything, and does not. The state was one HTTP request away the whole time.

The durable fix has a precedent in the same codebase. When you decline editor config writes, that choice persists in a small marker file in the workspace so the tool stops asking. Mode, port, and host should persist the same way, and restart should reuse the persisted config unless you explicitly override it. Until that lands, the banner should print the mode-complete command rather than the bare one, since it just talked to the daemon in order to produce the warning at all.

The demo below is the state ledger for that restart. Watch which pieces of configuration survive the round trip and which quietly reset to defaults.

Now the warning itself.

That check exists for a genuine problem. Most of what you see from vectr is rendered by the daemon, not the CLI, and the CLI imports the working tree fresh on every invocation. Upgrade the source while a daemon from last week is still running and you get a silent split: new CLI, old rendering, no error anywhere. I hit that in early July and could not see any of my own UX changes. The fix was a shared stamp, computed identically on both sides: the package version plus the short git sha when running from a checkout, stamped into the daemon at startup and exposed in its status response.

Then it started lying to me. The banner said the daemon was running older code, and printed 1.7.0+<sha> for the daemon against 1.6.0+<sha> for the CLI. Same seven characters of sha on both sides. The daemon's version was the higher one. Every part of that sentence contradicts the other parts.

Here is what happened. The version component comes from installed package metadata, and vectr is installed in editable mode, which means the metadata was written once when I ran the install and has been sitting there ever since. The project config had moved to 1.7.0. The metadata still said 1.6.0. The daemon, started after a reinstall, had picked up the newer metadata. So the CLI compared its own stale label against a fresher one and reported the difference as the daemon being behind.

Write the predicate out and the error is obvious:

So the fix is to compare the sha and to word the message as a version mismatch whenever only the label differs. Never claim "older" from a lexical difference.

What makes this worth more than a paragraph is the second-order damage. A false alarm does not cost you one wasted minute. It costs you the alarm. I now have a check that fires on a condition I know to be benign, which trains me to skim past it, which means it will not work the day it is right. And it caused real harm through a path nobody designed: it told me to restart, and the restart was the first bug. A false positive in a warning is not a cosmetic defect when the remediation attached to it is expensive.

Same restart, different casualty. The editor session I was working in lost every vectr tool. Not erroring when called, which would at least have been a clue: simply absent from the session's tool registry, as though the server had never offered them. They never came back. A new session in the same editor had all of them.

Meanwhile the daemon was completely healthy. I could hit its HTTP API by hand and get correct answers to the same questions the vanished tools would have answered. The server was fine, and what had broken was the relationship between it and a client that no longer existed to be told about it.

One mechanism behind this got caught and fixed on 29 July, and it is a good example of how ordinary the cause of a total outage can be. Vectr picks its port through a small registry: when a workspace has a previous port recorded, reuse it, specifically so that already-written editor config files stay valid. The reuse check probes whether the port is free by trying to bind it. A socket that was just closed sits in TIME_WAIT for a while afterwards, and a plain bind attempt against a TIME_WAIT socket fails. So the probe reported the previous port as busy, in exactly the window that every restart creates, and the port walk moved the daemon to 8766 while every config file on disk still said 8765.

Reproduced live: stop, start one second later, daemon bound to 8766. The editor hooks kept working the whole time, because they look the port up in a registry file instead of trusting a config value, which is why the outage looked partial and confusing rather than total. The tool surface broke with no error anywhere naming the cause. A second defect compounded it: the config writer reported that it had updated the settings file for every folder, and the file still read localhost:8765, because the port handed to the writer was not the port that ended up bound.

The fix landed as four changes, all deterministic:

• The free-port probe sets SO_REUSEADDR, matching what the actual server bind does, so TIME_WAIT reads as free.
• Port selection retries the previous port a few times with a short delay before it ever walks upward.
• Vectr's own entry in an editor config file is always rewritten to the port that was actually bound, while every other key in that file stays merge-only-add.
• Start and restart compare each known config file against the real bind and print a warning naming the stale files.

And now the honest part, because the port fix does not close this. The client I use does not re-handshake after it loses a connection. Nothing in the transport obliges it to, and other clients may well retry, but from mine the tools stay gone for the rest of that session even with the port preserved, because the process it completed its handshake with no longer exists. Restarting a server is cheap for the server and expensive for whoever was mid-sentence with it.

The detail I keep coming back to is that I did not know the HTTP fallback was usable until the moment I needed it, and I wrote the thing. An undocumented escape hatch is an escape hatch for exactly one person.

This one is older than the others, from mid-June, and it belongs here because it is the purest example of a test suite validating a fiction.

Symptom: the HTTP endpoint that answers "where is this symbol defined" returned a 500 for every symbol, in every language, every time. Not a rare path or an unlucky input, the whole endpoint, dead on arrival for as long as it had existed. The test suite was green.

Cause, in one line: the service method returns a LocateResult wrapper object, and the route iterated it as if it were a list.

# before
symbols = svc.locate_with_snippets(body.name, limit=body.limit)
...
    for s in symbols        # TypeError: 'LocateResult' object is not iterable

# after
result = svc.locate_with_snippets(body.name, limit=body.limit)
...
    for s in result.symbols

Now the part that matters. Why did nothing catch it?

Two reasons, and they reinforce each other. First, there were no tests for that route at all. The MCP path to the same feature was tested and happens to be unaffected, because it hands the whole result object to a formatter instead of iterating it. Somewhere along the way "the feature is tested" quietly became "every way of reaching the feature is tested," which are different claims once two translation layers sit over one core.

Second, the shared service mock that every API test uses was configured like this:

# before: a type the real service never returns
svc.locate_with_snippets.return_value = []
svc.format_locate.return_value = "No results."

# after: the real wrapper, with a real symbol inside it
svc.locate_with_snippets.return_value = LocateResult(
    symbols=[Symbol(name="PyDict_New", kind="function",
                    file_path="Objects/dictobject.c",
                    start_line=812, end_line=824, ...)],
    resolution_strategy="exact", query="PyDict_New",
)

A bare list. Convenient, empty, and a shape the real function has never returned in its life. Any test that had touched the route would have iterated that list happily and passed, because iterating a list is exactly what the buggy code does correctly.

The obvious objection is that Python's mock library will build a mock from the real object for you, and that specced mocks are the standard answer to exactly this. Worth being precise about what that buys: auto-speccing constrains which attributes exist and what call signatures are legal, so it catches you calling a method that was renamed or passing an argument that does not exist. It says nothing whatsoever about return types. return_value remains whatever you assign, and no framework I know of will tell you that the real function has never once returned that shape. Which leaves one reliable discipline: capture a real payload once, and build the fixture from it.

It happened again three days later in a different layer, which is what convinced me this was a class rather than an incident. A benchmark harness reported that every one of its hook injections had delivered zero content. That reads as a serious product failure, so I spent an afternoon investigating a product that was working fine. The metric was the broken part. The real event emitted by the agent CLI carries the hook's output as a JSON string inside a field, and the parser only walked nested dictionaries, so it never looked inside the string and therefore always found nothing. It had been green the entire time because the test mock fabricated a nested-dictionary shape the CLI does not emit. Injection had in fact worked: 7,404 characters delivered live. What it cost was a full paid benchmark run whose headline number was fiction.

Three rules came out of that pair, and they have held up:

• Mirror the real return type, not a convenient stand-in. If you are mocking a wire format, capture one real payload first and build the mock from it.
• Every externally reachable route gets its own test. Green on one transport proves nothing about another when the adapters are separate code.
• When a metric reads zero on a path you believe works, suspect the measurement before the product. A measurement bug looks exactly like the failure it is measuring, and it is cheaper to check.

I run coding subagents in isolated git worktrees. Two integration tests failed in those worktrees and passed on the main checkout. The failures looked like a regression from whatever the agent had just changed. They were not.

The tell was the clock. The failing test reported recall of 0.00 in about a third of a second, where a passing run takes fifteen. Nothing that searches a real index comes back in 0.34s. Nothing had been indexed at all, so precision and recall were trivially zero, and the test dutifully reported a search-quality score instead of the far more useful fact that its input was empty.

My first explanation was a good one, which is exactly how it became lore. Those worktrees live under a dot-prefixed directory, and the indexer prunes hidden directories during its walk:

dirnames[:] = [d for d in dirnames
               if d not in all_excluded and not d.startswith(".")]

Skipping hidden directories is correct: it keeps version control internals, virtual environments, and caches out of a code index. And it does explain that particular case. As the general rule it got treated as, it turned out to be wrong, which I only found out by bothering to run a control.

A three-way control run knocked out the hiddenness story as the general cause. The replacement hypothesis, "worktrees are the problem," was confounded in a way I find genuinely funny in retrospect: both non-hidden control worktrees had been created under /tmp, which macOS resolves to /private/tmp, and tmp is an unanchored entry in vectr's own ignore file. Every supposedly clean control was excluded too, by a different mechanism, producing an identical symptom. The decisive probe was a worktree at a path with no tmp component and no dot-directory: three tests passed in 16.23 seconds.

Underneath the folklore was a real product bug, nastier than the test failure that led me to it. The function deciding whether to index a file was passing the absolute path to the ignore-pattern matcher, while the two checks directly above it correctly used the workspace-relative path.

That matters because of what an ignore pattern means. A pattern in an ignore file is defined relative to the directory containing that file. An unanchored entry like tmp/ matches a directory named tmp at any depth below that root, and by definition it can never say anything about what sits above the root, because the file that declares it has no jurisdiction there. Hand the matcher an absolute path and you have quietly extended its jurisdiction to your entire filesystem, so the pattern starts matching ancestors that the repo owner never had an opinion about.

Fixed on 27 July at both call sites, the bulk walk and the file watcher, which now pass the relative path to the shared predicate. There was an intended side effect: path-scoped patterns like docs/* now match workspace-relative paths the way ignore-file semantics say they should. Before the fix they could never match anything.

The remaining work is not in the product, it is in the test. A test whose precondition is "the index is not empty" should assert that precondition and skip with a reason that names the matched pattern and the offending path component, rather than reporting recall 0.00 and inviting the next person to spend a control run rediscovering this. Environmental preconditions deserve the same care as assertions. If they stay implicit, the first plausible story about a failure becomes team knowledge, and team knowledge is very hard to unlearn.

Long agent runs kept dying mid-stream. The error said the stream had stalled or the connection had failed, so I filed it under network flakiness on the API side and re-ran things. Roughly forty kills accumulated across four days that way.

Then I did the thing I should have done on kill number two: lined the timestamps up against the machine's own power log.

pmset -g log | grep -E 'Wake|DarkWake'

Every kill matched a wake event to the second. Five or more exact matches in a row, and every one of the forty consistent with the pattern. That is not correlation you argue with.

The mechanism is unglamorous. Idle sleep on battery, or closing the lid, suspends the entire process tree. The connection to the API dies while everything is frozen. On wake, the ten-minute stream watchdog notices immediately that nothing has arrived and records a kill. The stack was never broken. It was asleep, and it got blamed for being unreachable while it was.

Practical resolution: caffeinate -is for the duration of any long run. The -i is the flag doing the work here, since it blocks idle sleep, which is the one that fires while you are away from the machine; -s blocks system sleep and only has effect on AC power. Neither can override closing the lid on battery, so it comes down to lid open or plugged in. On AC there is a system setting that prevents automatic sleeping when the display is off, and that is the durable version of the fix. One useful non-symptom: display sleep alone is harmless, so "the screen went dark and the run died" is a misleading intuition. Only system sleep kills.

The second half of this episode is about load rather than sleep, and it is why full-mode indexing counts as a hazard back in episode one. During one corpus re-index I watched the daemon hold 329% CPU for about two and a half hours, with system load at 21.24 and swap at 19.0 of 20.0 GB on a machine with 16 GB of RAM. Nothing else on the laptop was usable for the duration. The cost sits entirely in the search-index lane: roughly 350,000 embeddings for that corpus, split between one vector per chunk of code and a second pass producing a vector per symbol from its signature and docstring. Memory mode costs close to nothing by comparison, which is why the always-on instance runs unnoticed for weeks.

One aside that cost me an hour of confusion: processes launched from the editor's terminal are children of the editor's process tree, so the activity monitor attributes their memory to the editor. A 32 GB reading next to the editor's name was my own Python stack, not the editor being a memory hog.

Before the obvious suggestion: simply running the indexer at a lower scheduling priority does not fix this. Priority governs who gets the CPU, and by the time swap is at 19 of 20 GB the contended resource is memory, where being polite about CPU buys you nothing. A governor has to pace the work and cap concurrent batches, not just ask nicely for fewer cycles.

None of this is a vectr bug in the narrow sense. It became product work anyway, because the version a user hits is strictly worse than the version I hit: they install the tool, it starts indexing, their machine becomes unusable, and they uninstall. Three items came out of it. An index resource governor with a CPU budget, batch pacing, and pause-resume. Deferring the second embedding pass on large repositories. And the strategic one, that memory-only should be the default install, with search indexing an explicit per-workspace opt-in that shows a cost estimate before it starts. The entry price of a memory tool should be close to zero.

Laid out together, these are not six unrelated mistakes. Each one sits on a specific boundary between the program and something a test harness constructs, replaces, or ignores.

Three of these are not "the code is wrong" in the usual sense. The restart does what its arguments say. The ignore matcher matches the string it is given. The daemon binds a port that is genuinely free. In each case the defect is a broken contract with something outside the process, and the code inside the process is a faithful implementation of the wrong agreement.

Which suggests a practical filter for where to look. Ask what your tests replace with a stand-in, and what they construct fresh every run. Both lists are lists of blind spots. Mine were: the environment the daemon was started in, the identity of the code, the client on the other end, the service behind the mock, the absolute path of the workspace, and the operating system. Every bug in this post is on that list, and I could have written the list before finding a single one of them.

Six good catches makes a persuasive post, so here is the other column, because a method whose limits you cannot state is not a method.

I am the worst possible user. I know every workaround before I reach the wall that needs it, and I never read the documentation because I wrote it. More to the point, I have not experienced a first install in months, and the first ten minutes is where most people decide whether a tool is worth keeping. The famous version of this bias is teams on fast machines missing the performance problems ordinary users hit daily. Mine runs the other way: my laptop is underpowered relative to the corpora I index, so I feel resource pain acutely and am completely blind to the fresh-start experience I never repeat.

One machine, one operating system, one workflow. All of the above is macOS, on a single multi-folder workspace with one particular ignore file in it. Residency samples very deeply from one point in the space and says nothing at all about the shape of the distribution around it.

It finds classes, not rates. I can tell you the mock-fidelity failure exists and reproduces trivially. I cannot tell you how often users hit any of this, and any number I offered you would be made up. Rates need telemetry or a user base, and I have neither at a scale where a rate would mean anything.

The suite is not the villain here. It was green through all six of these and it is still the only reason I can change any of this without fear. The three instruments answer different questions. A test pins behavior you already understand so it stops moving. A reviewer interrogates the code you wrote. Residency is the only one of them that samples the assumptions nobody ever wrote down, which is also why it cannot be scheduled or assigned. Lean on residency alone and you land in the same place as leaning on tests alone: excellent coverage of one region, confident silence about everything else.

There is also a trap specific to this setup, which is that vectr is the memory of the sessions that build vectr. When the memory layer misbehaves, it degrades the context of the work fixing it. That is a real coupling and it is not all upside. It does mean I notice immediately. It also means my judgment about severity is made by someone whose current session state is affected by the bug, which is not a neutral vantage point. I have not found a good answer to that beyond writing findings down before acting on them.

None of the above matters if you route around it.

The rule is simple to state: a bug found while dogfooding is a product task. Fix it, or write it down with the end-user impact named. Never silently work around it. I was told this three times before it stuck, and I want to be precise about why it took three.

The first two times I agreed with it in principle and kept doing what everyone does, which is notice a rough edge mid-task, work around it in two seconds, and carry on with the thing I was actually doing. The workaround is always cheaper right now. That is the whole problem: the cost is real but deferred and lands on someone else. The third time it came with the sentence that made it stick. Roughly: it is not only about you, users are going to hit this too.

Which reframed the workaround as a decision rather than an omission. If I know the tool tells you to run a command that will melt your laptop, and I quietly stop running that command, I have not avoided a bug. I have decided that everyone who does not know what I know should hit it.

Mechanically, what makes it stick for me is three things. A place to put findings that is not my memory, with the user impact spelled out rather than a one-line reminder I will not understand next week. A default of fix-first for anything on the surface a new user meets in their first ten minutes. And a release rule that the tail does not begin while the discovered-issue list has items on it, which stops "we will get to it" from being a decision made by silence.

Two of the six bugs here are still open items rather than merged fixes, and I would rather say that than imply a clean six-for-six. They are written down with their impact, which is the deal. The distinction that matters is not fixed versus unfixed. It is filed versus forgotten.