Building Vectr — Part 1 of 3

If you're a human engineer navigating an unfamiliar codebase, here's what you probably do: you ask someone who knows it, or you grep for the error message, or you open the entry point and follow imports until you find the thing. Your brain does semantic compression the whole way — building a model of the system, discarding noise, following intuitions about where complexity tends to live. By the time you've read 20 files, you have a rough map that persists across days and sessions.

An AI code editor has the same tools — read files, run shell commands, grep — but completely different economics. Every Read call costs tokens. Every Bash call for grep costs a turn. Unlike a human who can skim-read at 1,000 words per minute and discard irrelevant content almost for free, an AI editor pays full price for every character it reads: it sits in the context window whether or not it was useful. Read the wrong 500-line file and you've burned context that could have held the answer.

The result, on unfamiliar codebases, is what I started calling the re-discovery tax: a cluster of navigation calls at the start of every session, before any actual implementation begins, spent on figuring out where things are. And because AI editors have no persistent memory between sessions, they pay this tax again and again — every session, on the same codebase.

In benchmarks I ran against real open-source codebases (more detail in Part 3), the re-discovery tax on CPython internals ranged from 6 to 23 tool calls per task before the first file write. Some sessions spent more turns navigating than implementing.

Before explaining what I built, I want to be precise about where grep breaks down — because "just use grep" is the natural reaction, and it's not obviously wrong until you try to use it systematically on unfamiliar code.

grep is a brilliant tool for confirming hypotheses you already have. If you know what you're looking for, it's nearly perfect. The problem is the case that isn't really an edge case: you don't know what you're looking for.

Say you're trying to understand how a Django application validates incoming JSON payloads before they hit the ORM layer. You might grep for validate. You'll get 200 results across 40 files — field validators, form validators, configuration validators, test fixtures. None of them are obviously the thing you want. You grep for json.loads. You get 30 results. You grep for request.data. That gets you closer, maybe. But you spent four greps and 15 minutes before you found the right file.

The deeper problem: grep requires you to already have a mental model of the codebase's naming conventions. An AI editor running on an unfamiliar codebase doesn't know whether payload validation is called validate_payload, check_request, parse_input, or _pre_process. It can guess, run four greps, and maybe find it. Or it can read files top-down until it finds something relevant. Both strategies are expensive in tokens and turns.

That's the core idea. But making it work on real code — not toy examples — requires solving several problems that aren't obvious from the outside. The rest of this post is about those problems and the design choices that resolve them.

Prose text has a natural unit of meaning: the paragraph. You can split a Wikipedia article into 200-word chunks, embed each one, and get a reasonable search system. Code doesn't work this way.

The standard naive approach for code indexing is the same line-window strategy borrowed from document search: take a sliding window of N lines with M lines of overlap, create a chunk, embed it, move the window. A common default might be 150-line windows with 50 lines of overlap. Simple, language-agnostic, works on any file format.

The problem is what happens at the window boundaries. Consider this function:

If a 150-line window happens to cut through this function — say, the signature and first few lines fall in chunk N and the body falls in chunk N+1 — neither chunk is independently meaningful. The chunk with just the body is missing the parameter names and return type. The chunk with just the signature has no implementation context. The embedding of a half-function is significantly worse than the embedding of the complete thing — it cannot capture what the function actually does, because it doesn't have the full picture.

The fix is obvious in retrospect: split at semantic boundaries. Functions should be complete units. Classes should contain their methods, or each method should be its own chunk with the class header prepended for context. The fundamental question — "what is the smallest independently meaningful unit of code in this language?" — has a clear answer: it's the function or method. But extracting those boundaries programmatically requires actually parsing the code, not pattern-matching on indentation or blank lines.

tree-sitter is a parser library that produces concrete syntax trees for source code — richer than what people sometimes loosely call an AST, but the key property is the same: every construct in the language has a named node with exact byte boundaries in the source. Unlike a regex-based approach that pattern-matches common function signatures, tree-sitter actually parses the grammar of each language and handles edge cases correctly: nested functions, decorators on multiple lines, multiline function signatures, arrow functions in JavaScript, generic bounds in Rust, anonymous classes in Java.

For each supported language, tree-sitter grammars define the language's syntactic structure as a formal grammar. At indexing time, Vectr runs a language-appropriate query against each file's syntax tree to find all function and class definitions. For Python, the query looks like this:

This query matches any function or class definition anywhere in the file — at the module level, inside a class, inside another function — and captures the name, parameters, and body as named nodes with precise byte-range positions in the source. You can then slice the original source file at those byte positions to extract complete, syntactically valid chunks.

The practical result: instead of chunks that contain partial functions and random intermediate lines, every chunk is a syntactically complete, independently meaningful unit of code. For classes, Vectr attaches the full class signature — including the base class list captured by @bases — as a header to each method chunk. So the chunk for WorkspaceLock.acquire() includes not just the class name but its inheritance context, which matters for search: a method of AuthenticatedView(LoginRequiredMixin, View) has a meaningfully different semantic context than a method of a plain View.

What about languages without tree-sitter grammars?

tree-sitter has grammars for most major programming languages: Python, JavaScript, TypeScript, Java, Go, Rust, C, C++, Ruby, PHP, and many more. For languages where grammars aren't available — rare proprietary DSLs, some configuration formats that don't parse as a programming language — Vectr falls back to a line-window approach with larger windows and more overlap. The quality is lower, but coverage is maintained. In practice, the fallback is invoked for fewer than 2% of files in any codebase I've indexed.

This chunking strategy — complete syntactic units, class headers attached to method chunks, fallback for unparseable languages — is the single biggest quality improvement in Vectr's indexing pipeline compared to a naive line-window baseline. Everything downstream (embedding quality, search relevance, working memory precision) depends on the chunks being meaningful units.

Not all embedding models are equally good at code. Models trained primarily on prose text — blog posts, books, Wikipedia — have learned representations of natural language semantics. They encode meaning in terms of topics, concepts, and linguistic relationships. Code has different regularities: symbol names, type signatures, control flow patterns, API call chains.

A general-purpose embedding model will encode def authenticate_user(token: str) -> Optional[User] and "user login validation" somewhat close together in vector space — but not as close as a model trained on millions of similar functions, which has learned the specific patterns that code uses to express authentication logic. The difference in retrieval quality on code search benchmarks is significant: code-aware models routinely outperform general-purpose models by 10–20% on tasks like "find the function that handles X."

Vectr uses Snowflake/snowflake-arctic-embed-m-v1.5, a 110-million-parameter model that produces 768-dimensional embedding vectors and runs in under 100ms per batch on a modern laptop CPU. It is optimized for retrieval tasks across both natural language and code, and its size keeps local inference practical without sacrificing too much quality.

The tradeoff is a one-time setup cost: the model weighs roughly 440MB and needs to be downloaded on first run. This is a real friction point — some users hit it and bounce. It's a known issue. The alternative (mandating API access) would exclude a large fraction of the most valuable use cases.

One detail worth noting about how the embedding is structured: the query and each code chunk are embedded in asymmetric modes. Queries use a Represent this query for searching relevant code: prefix, while code chunks are embedded with an Represent this code snippet: prefix. This is a property of arctic-embed-m's training regime — the model is a single encoder (not two separate models), but it was trained with different input prefixes for query-side and document-side inputs so that the resulting vectors occupy compatible but distinct regions of the same embedding space. Using the wrong prefix reduces the cosine similarity between semantically related query-chunk pairs — the vectors for "user authentication" and verify_token end up further apart in embedding space than they should be. Using the wrong mode measurably degrades retrieval quality.

Vector search finds chunks that are semantically similar to your query. It handles natural language queries ("JWT validation"), concept queries ("connection pooling logic"), and paraphrases well. But it has a well-known weakness: it handles exact symbol names poorly.

If you search for _handle_workspace_lock_conflict — an exact function name — a vector search might not rank it first. The embedding of that function name is just one point in a crowded neighborhood of similar-looking function names in the vector space. They all compete, and the margin between them is small. Meanwhile, BM25 will find it immediately — it's a keyword search algorithm, and exact string matches get the highest possible score.

The inverse is also true: BM25 cannot find "retry logic with exponential backoff" if the function is called _schedule_attempt_with_delay and its docstring says nothing about backoff. Zero keyword overlap means zero BM25 score. Vector search finds it because the semantic cluster it belongs to is close to the query in embedding space.

The right system uses both. Every query in Vectr runs both a vector search and a BM25 search in parallel, then combines the two ranked lists using a weighted formula. The weight assigned to each approach depends on a characterization of the codebase and query context:

This combination isn't a magic formula. It's an approximation that works better than either approach alone in practice. The benchmark on the Apache Camel codebase (58,000+ Java files) showed a 73% reduction in Read+Bash navigation calls compared to the baseline AI editor with no index. A purely semantic search would have missed exact class names that had already been established as part of the task context. A purely keyword search would have missed the conceptual queries that came next. The specific weights in the table above are the values actually used in Vectr's implementation — they are a starting point, not a theoretically derived optimum. I tuned them against the benchmark dataset and they haven't been revisited since.

The codebase characterization heuristic — large + unfamiliar = semantic-heavy, small + familiar = BM25-heavy — is computed from file count, total token count, and how recently the index was last queried. It's a proxy for "how likely is it that the user knows the naming conventions?" A better signal would come from explicit feedback (did the user click through on the result?), but that requires session tracking that I haven't built yet.

Semantic search and BM25 handle "find me the code for this concept" well. But there's a different navigation pattern that neither handles: "find me everything that calls this function."

Call graphs are a fundamentally different kind of knowledge. They describe relationships between code entities — which function calls which, which class imports which module, which HTTP route maps to which handler. You can't answer "who calls process_workspace_changes?" with a text search, because the callers don't contain that function's body — they just contain a call to it by name, somewhere in their own bodies.

Vectr builds a symbol graph during indexing. For each file, tree-sitter extracts four kinds of information:

What the symbol graph captures

The resulting graph enables exact lookups that are categorically different from text search. vectr_locate("WorkspaceLock") returns a file path and line number in under 10ms — no embedding, no ranking, pure symbol table lookup. vectr_trace("acquire_lock") returns all callers and all callees in one round-trip. These are not search results — they are graph traversals, and they produce exact answers rather than relevance rankings.

HTTP route indexing deserves a note: treating GET /api/workspace/{id}/status as a first-class named symbol means an AI editor can navigate directly from an API endpoint to its handler — a common navigation need that would otherwise require grepping for the route string across all route definition files.

In practice, users don't always know the exact symbol name. Maybe you know it as WorkspaceLockManager but the actual class is WorkspaceLock. Maybe you know the module but not the fully qualified name. vectr_locate runs six fallback strategies in sequence, stopping at the first match:

Each strategy produces a LocateResult with a resolution_strategy field, so the caller knows how confident the match is. An exact match means you can act on the result immediately. A fuzzy match with edit distance 2 means you should verify before relying on it. This distinction matters in practice — a silent wrong navigation is worse than no navigation at all.

The first time you run vectr start on a large codebase, indexing takes time. CPython's 4,000+ Python and C files: about 8 minutes on a modern laptop. Django's ~1,800 Python files: about 2 minutes. Apache Camel's 58,000+ Java files: closer to 45 minutes.

You obviously can't re-index the entire codebase every session. The solution is incremental re-indexing with an mtime cache.

During initial indexing, Vectr writes a file at ~/.cache/vectr/{hash}/index_cache.json that stores the modification timestamp of every indexed file. On subsequent vectr start calls, Vectr checks each file's current mtime against the cached value. Only files that have changed since the last index run are re-indexed. On a typical active development session where you've modified 5–10 files, subsequent re-indexing takes under 5 seconds.

The {hash} in the cache path is a short SHA-256 hash of the absolute workspace root path — just enough characters to make two different paths produce different cache directories with overwhelming probability. Each workspace gets its own isolated cache directory, so you can run Vectr on multiple projects simultaneously without cross-contamination.

Handling deletions

The tricky part of incremental re-indexing is deletion. If a file is removed, it has no mtime to check against — it simply isn't there. Vectr handles this by also storing the complete set of indexed file paths in the cache. At startup, it diffs this set against the current file tree and removes all chunks belonging to deleted files before re-indexing any modified ones. The order matters: process deletions first, then updates, then new files. This prevents a renamed file from leaving orphaned chunks in the index.

The watchdog listener

During an active session, Vectr runs a watchdog listener on the workspace root. When a file is saved, the listener queues it for re-indexing in the background. You don't notice this happening — the next vectr_search call after a file save will return results that reflect the change. This closes the loop between editing and searching: you refactor a function, save, and subsequent searches return the updated version.

The re-indexing threshold is conservative: any write event to a source file not excluded by .vectrignore triggers re-indexing of that file's chunks, even for a one-line change. In practice, most editors issue multiple rapid write events per save (the editor writes, then a formatter rewrites, then the LSP may write again). The watchdog listener debounces these at 300ms — only the last write in a burst of events within 300ms actually triggers re-indexing. Without debouncing, a single save in a project using aggressive auto-formatting would trigger 3–5 redundant re-index operations. You'd rather spend 200ms re-indexing one file once than return stale results on a query that matters.

Not everything in a repository should be indexed. Generated files, binary assets, vendor directories, build outputs, test fixtures — these consume index space and add noise without contributing useful search results. A search for "authentication handler" should not return results from node_modules/ or minified vendor JavaScript.

Vectr reads a .vectrignore file from the workspace root using glob patterns. The syntax follows .gitignore conventions — trailing slash for directories, * for single-level wildcard, ** for recursive match (implemented via Python's pathlib.Path.match()) — but Vectr does not implement the full gitignore specification: the ! negation prefix is not supported. If you already have a well-tuned .gitignore, you can copy most of it directly. A typical .vectrignore for a mixed-stack project:

There's also a --exclude flag on vectr init for quick one-off exclusions without creating a file.

The practical importance of this: a codebase with node_modules/ in its tree will typically contain 5–20x more code from installed packages than from the project itself. Every search result page would be dominated by library code you didn't write and don't want to navigate. Excluding vendor directories is the single most impactful configuration change most users can make, and it should be done before the initial index run to avoid having to rebuild from scratch.

Let's make the full pipeline concrete. When an AI code editor calls vectr_search("workspace lock acquisition and release"), here is the complete sequence of operations:

The result for the query above might look like:

Instead of reading 15 files to find these three functions, the AI editor reads one search result. Navigation time drops from 15 calls to 1. Across a full task session, this compounds: the re-discovery tax on CPython internals dropped from 6–23 tool calls per task to 1–3 in the benchmarks I ran with the index active.

Every system has decisions that looked reasonable at design time and only reveal their cost in production. Here are the three I'd revisit.

The Python 3.14 requirement

Vectr currently requires Python 3.14. The core reason is that the codebase uses match/case structural pattern matching extensively, and some asyncio event loop patterns that technically work on 3.10+ but behave differently in ways we hit in practice. In retrospect, 3.11 would probably work with a few hours of refactoring. The 3.14 requirement has been the single biggest adoption friction — a significant fraction of engineers who encounter it close the tab rather than update their Python version. This is worth fixing.

ChromaDB as the vector store

ChromaDB — a vector store that handles embedding persistence and similarity search — works fine, but it's heavyweight for what we actually need. The full HNSW index with persistence, the Python client layer, and the inter-process communication overhead add about 200ms specifically to ChromaDB's startup contribution — not total Vectr startup, which is ~280ms including mtime diffing and watchdog initialization. For a tool invoked at the start of every session, that's noticeable. For v2, I'd consider a lighter in-process option — possibly a purpose-built flat index with ANN search for codebases under 100k chunks, falling back to HNSW only for larger ones.

The BM25 implementation

The rank-bm25 library is pure Python and fast enough for codebases under 50,000 chunks. Beyond that, it starts to show latency — the in-memory index grows large and query time climbs past what's acceptable for interactive search. The right long-term solution is probably integrating BM25 scoring directly into the vector store query pipeline, or using a tokenized inverted index instead of the full in-memory approach. For current use cases, where most codebases are under 20K chunks, it's fine. But it's a known scale ceiling.