I Deleted Every Ranking Heuristic From My Code Search Engine

I want to describe the pile fairly, because the point of this post is not that I was careless. Each of these was a locally sensible response to a real failing query. The retrieval pipeline underneath is a fairly standard one: a hybrid dense+BM25 first stage builds a candidate pool, a cross-encoder reranker re-scores it, and a symbol graph from tree-sitter handles exact name lookups. The heuristics all lived in the ranking layer, between retrieval and the final order.

Here is what accumulated, more or less in the order the failing queries arrived:

• Two curated stopword files — one for programming terms, one for English — filtering tokens out of the query before symbol matching.
• A doc-intent query classifier, is_doc_intent_query: prefix lists plus "contains any of these substrings" lists, deciding whether a query "wanted documentation," then gating pool reservations and score multipliers on the answer.
• Forced-inclusion machinery: if the query named a symbol token, force the matching chunks into the pool — governed by a short-verb allowlist that decided which tokens were "real" enough to count.
• A symbol-identity boost with a minimum-leaf-length constant, rewarding chunks whose name overlapped the query.
• A query_wants_tests branch that fired when the query looked test-flavored.
• And the connective tissue: assorted if/else in the final scoring pass to wire all of the above together.

Every one of these fixed the query in front of me. That is exactly what made them so easy to keep adding. But look at what they have in common, because it is the whole thesis of this post in one sentence: every single one is a branch on the words in the query. Stopwords branch on which tokens you typed. The doc-intent classifier branches on whether your phrasing contains certain substrings. Forced inclusion branches on whether you named a symbol. Each rule is, underneath, a bet that the next query will look like the last one I patched for.

I could feel it getting worse and told myself it was getting better, because the benchmark kept going green. What finally forced the issue wasn't a code smell. It was a single query I could not fix without breaking another one — and the moment I understood why, the whole approach fell over.

The query was "delete rows from database table." The right answer, for anyone working in Django, is QuerySet.delete — the method whose job is to delete the rows a queryset matches. In my pipeline it ranked 11. Sitting at the top, above it, was DatabaseCache.delete.

Now, DatabaseCache.delete is not a garbage result. It is the delete method of Django's database-backed cache — a cache that stores its entries as rows in a table, and whose delete therefore literally deletes rows from a database table. Read the query again. It is a perfect description of the wrong answer. If I wrote a keyword rule to demote it, I would be breaking the legitimate query from someone who actually wants the cache's delete. There is no phrasing test, no stopword, no boost that separates these two, because the thing that makes one canonical and the other a look-alike is not in the query at all.

This is the moment the pile of heuristics stopped being a maintenance annoyance and became an impossibility proof. Every tool I had built inspected the query. The signal I needed was categorically absent from the query. You cannot read information out of a place it was never written. I could keep patching individual queries forever and never touch the thing that actually decides this one.

So I went looking for the signal where it might actually live: in how the codebase refers to its own code.

The natural place to look for canonicality is the call graph. If QuerySet.delete is the canonical delete, surely the rest of the codebase calls it far more than it calls a niche cache's delete. Rank symbols by PageRank over the call graph and the canonical one should float to the top. That was the plan. It failed, and the way it failed is the important part.

My call extractor stored edges by bare leaf name. When it saw obj.delete(), it recorded an edge to delete — with no idea which class obj was. So every .delete() anywhere in Django, on any object, counted toward every method named delete. The graph physically could not tell QuerySet.delete apart from DatabaseCache.delete, because it had thrown away the one fact that distinguishes them.

The measurement is stark. Symbol-level PageRank over those leaf-only edges gave the two methods near-identical importance — 1864 against 1834. That is a statistical tie. I ran the same check on get, another badly overloaded name, and got 199 against 206. Neck and neck, both times.

The obvious next thought: resolve the receiver. Figure out that obj is a QuerySet and key the edge on QuerySet.delete. Cheap static rules get you some of the way — self and cls resolve to the enclosing class, x = ClassName() tells you x's type, Name.method() is explicit. I implemented them and measured the coverage: those rules resolved only 16% of 32,698 call edges in the package I tested.

And here is the cruel part — the 16% you can resolve is the wrong 16%. The calls that actually discriminate QuerySet.delete from the rest are the dynamic ones: some_qs.delete(), where some_qs came out of a function three call frames up. Those are precisely the calls cheap rules can't touch. In a dynamically typed language, the references that carry the signal are the references that resist static resolution. The cheap fix cleared the easy cases and left every hard case exactly where it was.

Before rebuilding, I made myself do the reading I should have done first. I wanted to know whether serious code-search systems solve canonicality with query-time cleverness or with something else entirely. The answer was consistent, and it was not the thing I had been doing.

Start with the empirical ceiling on the approach I'd been leaning on. The paper "Structural Code Search using Natural Language Queries" reports that translating a natural-language query into a structural search significantly outperforms baselines based on semantic code search, by up to 57% on F1, with their approach landing in the 55–70% precision/recall range. The reading I take from that: vector similarity alone under-serves queries with structural intent. Embeddings are necessary and they are not sufficient, and no amount of query-side patching changes what the embeddings can and can't encode.

Then the question of where precise structure comes from. Sourcegraph's SCIP, the SCIP Code Intelligence Protocol, is "a language-agnostic protocol for indexing source code, which can be used to power code navigation functionality such as Go to definition, Find references, and Find implementations." That is exactly the "who references what, precisely" that my leaf-only edges couldn't express. The lesson isn't "adopt SCIP"; it's that precise reference resolution is treated as an indexing primitive, computed once at build time, not as something you reconstruct per query.

The most direct precedent for what I ended up doing is Aider's repository map. Its write-up, "Building a better repository map with tree sitter," uses tree-sitter to find "where functions, classes, variables, types and other definitions occur" and "where else in the code these things are used or referenced," then runs "a graph ranking algorithm, computed on a graph where each source file is a node and edges connect files which have dependencies" to select the most important parts of the codebase. That is canonicality as a standing global prior (most-referenced is most-important), computed with zero inspection of any query.

And the broader wave points the same direction: systems like CodexGraph ("Bridging Large Language Models and Code Repositories via Code Graph Databases") treat the repository as a queryable graph rather than a bag of embedded chunks, so an agent can navigate structure directly.

The pattern across all of it is the same, and it is the opposite of query-time keyword logic: invest in structural signals computed at index time (precise reference resolution, reference-graph importance) and let neural relevance handle meaning. Nobody serious is ranking canonicality by inspecting the user's words. So I went to measure whether the structural signals would actually separate my two deletes.

The call graph had failed because it stored the wrong thing. But my symbol graph already knew, for every definition, where it lived and what referenced it. That is enough to build a def↔ref graph at two granularities the leaf-only call graph couldn't: files and classes. Both sidestep the exact thing that broke centrality — they never need to resolve some_qs to a class at ranking time.

File-level reference importance

First I ranked files by reference-graph importance, the Aider-style move, built from my existing symbol graph. Over the 2,297 source files in Django's reference graph, django/db/models/query.py, the home of both QuerySet.get and QuerySet.delete, ranked 5. The look-alikes' home files sat far below: the database cache at 96, the admin options at 147, the cache base at 28. The canonical file is near the very top of the corpus; the neighbors are buried tens to over a hundred places down. No query was consulted to produce any of these numbers — the ranking is identical whether you search for "delete rows" or anything else.

Owning-class importance

The sharper signal is at the class level, and it's the one that made me stop and re-run it to be sure. Rank classes by reference importance, then let each method inherit its class's rank. This dodges receiver resolution completely: I don't need to know that some_qs is a QuerySet to know that the QuerySet class is central and the DatabaseCache class is niche. Over Django's 1,709 classes, QuerySet ranked 70; DatabaseCache ranked 1434.

Then the actual test. Take every definition sharing a leaf name and order them by this class prior. For get, which has 31 same-leaf definitions across the corpus, QuerySet.get comes out #1. For delete, with 29 definitions, QuerySet.delete comes out #2 — behind only Model.delete, which is itself a completely legitimate canonical delete. That is the exact disambiguation the heuristics never achieved, produced by one global prior with no query in sight.

All the measured separations in one place, so you can see the jump from "tied" to "resolved" happen as the signal moves from leaf-blind to structure-aware:

Ordering every same-leaf definition by that class prior is what actually closes the case: get's 31 definitions put QuerySet.get first, and delete's 29 put QuerySet.delete second, behind only the equally-canonical Model.delete.

I had a choice at this point that I think is the real decision in any rewrite: run the old heuristics and the new priors side by side for a while, or delete first and rebuild clean. Running both is the cautious-sounding option, and it's a trap. The heuristics weren't just inert code — they were actively fighting the new signal, and their tests and documentation would keep telling the next person (including future me) that the pile was load-bearing. So I deleted.

Out went the two stopword files and the code that loaded them, the doc-intent classifier and its config block, the forced-inclusion machinery and its allowlist, the symbol-identity boost, the test-intent branch, and the if/else in the scoring pass that served all of them — along with their tests and the doc references that described them. 9 files, −3,817 lines, one commit.

Then the moment I'd been quietly dreading. I ran the full suite expecting a field of red — surely thousands of lines of ranking logic were holding something up. 1,326 tests passed, and the deterministic precision/recall gate stayed green. The regressions I'd braced for mostly didn't happen.

What I kept is as important as what I cut, and the line between them is the whole transferable idea. I did not delete everything that touches ranking. I kept a set of content-based signals: trivial-chunk demotion, so that a bare } can't outrank real code; exact-duplicate deduplication; code-aware BM25 tokenization; file-type classification for indexing. Every one of those describes the corpus — a property of the data that is true regardless of what anyone searches for. What I deleted was everything that inspected the query for keywords to branch on.

Notice what the color change is actually telling you. The rebuild didn't add cleverness — it removed a category of decision. The clean pipeline makes exactly zero branches on the user's words. It retrieves, reranks, and then applies priors that were computed at index time and are the same for every query on earth. That is not a smaller pile of heuristics. It's a different kind of thing.