Why AI Code Assistants Waste Your Context Window — and How RAG Fixes It

The first instinct when building a code assistant is to send as much context as possible. Your project has a utility module? Include it. There's a shared type definitions file? Include that too. If the model's context window is 128,000 tokens, fill it to the brim — more information has to be better, right?

This is called context window stuffing, and it has three distinct failure modes. Each one gets worse as the codebase grows. To understand why, you first need a precise model of how a transformer reads a prompt.

What the model actually receives

A transformer does not read a prompt sequentially, the way a human reads a page from left to right. Instead, it processes all tokens simultaneously, and every token attends to every other token in the sequence. The attention mechanism is the machine that computes how much each token should "look at" every other token when forming its representation.

The output of attention for a single token is a weighted average of all the other tokens' value vectors. The weights are computed by comparing the current token's query vector against every other token's key vector. When you add more tokens to the context, you are not adding more information to a receptive mind — you are adding more competitors for a fixed budget of attention weight.

The attention mechanism was introduced in the paper "Attention Is All You Need" (Vaswani et al., 2017). Its core computation is:

The notation QKT means: for each token, compute a dot product between its query vector and every other token's key vector. The dot product is large when two vectors point in the same direction (high relevance between the pair), and near zero when they are orthogonal (unrelated). Multiplying by the transposed key matrix K^T does all N×N such comparisons in a single matrix operation. The result is a matrix of raw relevance scores. Dividing by √dk prevents those scores from becoming so large that softmax saturates (all weight on one token).

The softmax step is the dilution mechanism. Because softmax always outputs a probability distribution — all values sum to exactly 1 — attention weights are a zero-sum resource. When there are N tokens in the context, the average attention weight is 1/N, regardless of what any individual token does. The total budget is fixed at 1.0.

This does not mean every token gets exactly equal attention — the model can still concentrate on a small subset of tokens if the dot-product scores make those tokens stand out sharply. But the key word is if. In a real codebase, most of the context is irrelevant to any particular completion request. Hundreds of unrelated function definitions produce hundreds of tokens with moderately non-zero dot products. These tokens collectively consume the majority of the softmax budget even though each individual one has low relevance. The useful signal — the few functions that actually matter — must compete against this crowd, and its share of the attention budget shrinks as N grows. Signals that were reliably attended to at N=2,000 tokens can fall below effective threshold at N=50,000 tokens.

Softmax sharpening and temperature

The scaling factor √dk in the formula is there specifically to control how "peaky" the attention distribution is. Without it, dot products in high-dimensional spaces get very large, pushing softmax into saturation — where almost all weight goes to the single highest-scoring token. The scaling keeps the distribution smooth enough for gradients to flow during training. But the same property means that with a very long sequence, the model is doing softmax over hundreds of thousands of values. The distribution becomes nearly uniform across irrelevant tokens, and the useful signal struggles to dominate.

Attention dilution is one problem. A second, independent problem compounds it: position bias. Modern language models do not attend to all positions in their context with equal reliability. They preferentially attend to tokens at the beginning and end of the sequence, and perform significantly worse on information placed in the middle.

This phenomenon was studied in a 2023 paper by Nelson Liu et al. titled Lost in the Middle: How Language Models Use Long Contexts. The researchers tested models on multi-document question answering, varying the position of the document containing the answer. The results were sharp: when the answer document was at position 1 (the very beginning) or the last position, accuracy was high. When it was at position 10 of 20 documents, accuracy dropped by more than 30 percentage points — even though the information was technically within the model's context window.

Why does this happen architecturally?

The cause is rooted in RoPE (Rotary Position Embeddings), the positional encoding scheme used in most modern language models. RoPE works by rotating the query and key vectors by angles that are proportional to their positions in the sequence. The key consequence: the dot product between a query at position m and a key at position n includes a term that depends on their relative distance (m−n). As that distance grows, the rotational difference introduces a decay in the inner product — semantically relevant tokens at distant positions must overcome this rotational penalty to receive high attention. Tokens near the beginning of the sequence are at low distance from almost every other token, giving them an inherent structural advantage. The model also learns, through training, to rely heavily on the most recent tokens for predicting the next one. The result: early tokens and the last few tokens both get reliable attention; the middle of a long context is structurally disadvantaged by both effects simultaneously.

A 2024 paper from the University of Washington, MIT, and Google (Found in the Middle) demonstrated that this bias can be partially corrected by calibrating attention weights at inference time — but this requires modifying the model's internals, something you cannot do when calling an API.

For practical purposes, if you are a user or a developer calling a commercial API, the position bias is a fixed environmental constraint. The implication is direct: you cannot mitigate it by stuffing more context. You can only mitigate it by ensuring the relevant content appears at the beginning of your context — which means retrieving it selectively rather than including everything and hoping.

Even if you were willing to accept degraded attention quality, there is a third reason not to stuff context: the compute cost of attention scales quadratically with sequence length.

To compute the full attention matrix, the model must compare every token's query against every other token's key. If your sequence has N tokens, this requires N × N comparisons — producing an N² attention matrix. Doubling the context length quadruples the compute required for attention. This is not a limitation of current hardware; it is a mathematical property of the full-attention transformer architecture.

In production, this means that a code assistant filling 100,000 tokens of context is not just 10× slower than one filling 10,000 tokens — it is closer to 100× more expensive in attention compute alone (before accounting for the rest of the forward pass). It is also 10× more expensive in the API pricing sense, since most APIs charge per input token. You are paying more to get worse results.

Retrieval-Augmented Generation reframes the problem. Instead of asking "how can we give the model the whole codebase?", it asks "how do we figure out which parts of the codebase are relevant to this specific completion request, and send only those?"

The answer has two phases. First, an offline indexing phase where the codebase is processed, divided into chunks, and each chunk is converted into a vector representation (an embedding) that captures its semantic meaning. These vectors are stored in an index optimized for fast similarity search. Second, an online retrieval phase that happens at query time: the developer's current context (cursor position, open file, recent changes) is converted into a query vector, and the most similar chunks from the index are retrieved and injected into the prompt.

The model then receives a context window that is not a random cross-section of the codebase — it is the small set of pieces most likely to be relevant to the task at hand. This keeps the context window small, focused, and full of signal rather than noise.

Each of these steps has engineering depth. The three that most affect output quality — and that most implementations get wrong — are chunking, retrieval strategy, and prompt injection order. The next sections cover each in detail.

The purpose of chunking is to divide the codebase into units that can be individually embedded and retrieved. The naive approach is fixed-size chunking: split every file every 256 tokens, regardless of where that lands in the code structure. It is easy to implement. It is almost always wrong for code.

Code has structure that text does not. A function is a unit of meaning. A class definition with its methods is a unit of meaning. Splitting in the middle of a function produces two chunks, each of which is semantically incomplete — one has the function signature and the early logic, the other has the return path. When embedded, neither chunk represents the function accurately. When retrieved, neither chunk gives the model what it needs to understand the function's contract.

Fixed-size chunking failure mode

Consider a Python function that is 80 lines long. With a 50-token chunk size, it gets split into chunks that might look like:

AST-based chunking

AST-based chunking uses a language parser — specifically tree-sitter — to parse each file and extract logical units. Instead of asking "where does the 256th token fall?", it asks "where are the function boundaries, class boundaries, and docstrings in this file?"

Each extracted unit becomes a chunk. A function definition — signature, body, docstring, and any inline comments — stays together. The chunk's metadata stores its file path, start line, end line, and the type of unit (function, class, method, module-level code). This metadata is as important as the chunk text itself: it tells the retrieval system where in the codebase this chunk lives, enabling it to surface related chunks from the same file.

Sliding-window augmentation

One practical addition: after AST chunking, each chunk is sometimes augmented with a small surrounding context — the preceding import block, the class it belongs to, or the file's module-level docstring. This gives the embedding model enough context to produce a vector that reflects not just the chunk's local meaning but its role in the larger structure. The key is that this surrounding context is used only for embedding, not retrieved as part of the chunk text. The retrieved text remains the original chunk.

Once the codebase is indexed, retrieval is the step that determines which chunks are surfaced. There are two fundamentally different ways to search a corpus of text, and code needs both simultaneously.

Dense retrieval (semantic search)

Dense retrieval converts both the query and each stored chunk into a numeric vector (an embedding), then finds chunks whose vectors are closest to the query vector by cosine similarity. This approach is powerful because it can match meaning even when the exact words differ. A query like "how do we handle rate limit errors?" will surface functions named throttle_on_429 or backoff_retry — code that addresses the same concern using completely different identifiers.

The embedding model used matters a great deal for code. General-purpose text embedding models (trained primarily on natural-language text) perform poorly on code because their training data underrepresents the structural and syntactic conventions of programming languages. Code-specialized embedding models — such as voyage-code-3 or OpenAI's text-embedding-3-large trained with code data — produce substantially better representations for function bodies, type signatures, and API calls.

Sparse retrieval (keyword search / BM25)

BM25 is a classical information retrieval algorithm that scores documents by how well they match the exact terms in a query. It does not understand meaning; it counts words. But for code, exact keyword matching is often exactly what you want.

If the developer is working on a bug involving PaymentGateway.process_refund, a dense embedding search might return several semantically related functions — but the exact function they need might not score highest on semantic similarity. A BM25 search for the exact string process_refund will find it immediately. Similarly, error codes, configuration key names, and exact API method names are better retrieved lexically than semantically.

Query: "process_refund implementation" — neither method alone captures all the relevant chunks.

Running both BM25 and dense retrieval solves the coverage problem — you surface both the exact-match candidates and the semantically similar ones. But now you have two ranked lists and need to combine them into a single ranked list to pass to the model. This is the rank fusion problem.

The challenge is that BM25 scores and cosine similarity scores live in completely different numerical ranges. BM25 produces scores that depend on corpus statistics (document length, term frequency, corpus size). Cosine similarity is bounded in [-1, 1]. You cannot add or average them directly in a meaningful way.

Reciprocal Rank Fusion (RRF)

RRF avoids the normalization problem entirely by ignoring raw scores and working only with ranks. The word "reciprocal" means 1/x — the score assigned to a document is the reciprocal of its rank in each list. A document ranked 1st gets a high score (1/61 with k=60); a document ranked 50th gets a low score (1/110). For each candidate chunk, the combined RRF score sums these reciprocals across all ranked lists:

The effect of the formula: a document that ranks 1st in the BM25 list and 1st in the dense list will have an RRF score of 1/(60+1) + 1/(60+1) ≈ 0.033. A document that ranks 1st in one list but 100th in the other will score 1/(60+1) + 1/(60+100) ≈ 0.022. Consistently high-ranked documents across multiple sources float to the top. Documents that are strong in only one source get discounted. This is the behavior you want: candidates that both BM25 and semantic search agree on are the most reliably relevant.

When not to use RRF

RRF treats all ranked lists as equally authoritative. In practice you might know that for a particular query type — say, looking up a config constant by its exact name — BM25 should be weighted more heavily. Some systems implement weighted linear combination of normalized scores instead of RRF, using query classification to choose weights. This adds complexity but can improve precision for domain-specific query patterns.

After hybrid search and RRF, you have a list of, say, 20 candidate chunks. You want to inject only the top 3–5 into the prompt. The question is whether the RRF ranking is good enough to trust for this final cut, or whether a second, more expensive sorting pass is worth running.

This is where cross-encoder rerankers come in. The retrieval methods used so far — both BM25 and dense embedding search — are bi-encoders: they encode the query and each document independently, then compare the resulting vectors. This is fast because the document embeddings are precomputed. But it means the query and document never interact during encoding — the model cannot see the query while deciding what aspects of the document to emphasize.

A cross-encoder takes both the query and a candidate chunk as a single concatenated input and produces a relevance score. Because both texts go through the model at the same time, the model can attend to relationships between the query and the document that a bi-encoder cannot. The accuracy improvement is often significant — but the cost is that you cannot precompute: the cross-encoder must run on every query-candidate pair at inference time.

The practical solution is a two-stage architecture: use the fast bi-encoder pipeline (dense + BM25 + RRF) to get the top 20 candidates, then run a cross-encoder on only those 20 to get a final ranking. The cross-encoder sees a small, manageable candidate set and produces a high-quality relevance ordering. The top 5 from this final ranking go into the prompt.

Cursor is the most architecturally transparent commercial code assistant. Its codebase indexing implementation has been publicly described in enough detail to serve as a concrete reference for the pipeline above.

Indexing

When you open a project in Cursor, it begins indexing the codebase in the background. Files are chunked locally on your machine. Chunks are then sent to Cursor's servers, where they are passed through an embedding model — either OpenAI's embedding API or a custom-trained model, depending on the feature context. The resulting vectors are stored in Turbopuffer, Cursor's vector store of choice. Metadata — file path, line ranges, and a hash of the chunk content — is stored alongside each vector. File paths are obfuscated client-side before any data leaves your machine.

Cursor caches embeddings by chunk hash. The second time you index the same codebase (or if most files haven't changed), it skips recomputing embeddings for unchanged chunks, making incremental re-indexing fast.

Query signal construction

The query is not just "what the developer typed." Cursor monitors the active cursor position and constructs a composite signal from several sources: the current file's surrounding code at the cursor position, any open editor tabs (which it weights as likely-related files), and recent edit history. This composite signal is embedded into a query vector.

Retrieval and injection

The query vector is sent to Turbopuffer, which performs ANN search, returning the top-k most similar chunk vectors. Cursor's client receives the result with obfuscated file paths and line ranges, then reads the actual code from the local filesystem. The retrieved chunks are injected into the prompt sent to the LLM. The model never directly touches the vector database — it only sees the retrieved text.

GitHub Copilot's approach

GitHub Copilot's context construction is less publicly documented but follows a similar pattern. For inline completion, it uses the current file content around the cursor (the prefix and suffix of the current file) plus a Jaccard similarity heuristic to find other open tabs that share significant token overlap with the current file. The @workspace symbol in VS Code triggers a more thorough indexing-based search of the workspace, analogous to Cursor's @Codebase.

The key distinction: Copilot's default inline completion mode is a fast, low-latency path that does not run full vector retrieval on every keystroke. Full retrieval is reserved for explicit chat interactions. This is a deliberate latency-accuracy tradeoff — inline completion needs to respond in under 100ms to feel responsive; full RAG retrieval adds 200–500ms of overhead.

RAG is not a complete solution. Understanding its failure modes is essential for building a system that handles them gracefully.

The "big context window makes RAG obsolete" argument

As context windows grow to 1M and 10M tokens (Llama 4 Scout reached 10M tokens in 2025), a natural question arises: at some point, can we simply include the entire codebase in context and skip RAG entirely?

The answer, in practice, is no — for three reasons. First, even a modest codebase of 200,000 lines of Python easily exceeds 2 million tokens. Most production monorepos are far larger. Context windows are growing, but so are codebases. Second, as we established in Sections 2 and 3, attention quality degrades with length regardless of the window's nominal size limit. A 2M-token context window does not mean 2M tokens of equal-quality attention. Third, latency and cost penalties remain. A 1M-token prompt is extraordinarily expensive and slow to process, even on the best hardware. For interactive coding assistance that must respond in under a second, this is impractical.

The most defensible position: large context windows and RAG are complementary tools, not alternatives. RAG determines what to include; the context window determines how much you can include once you've been selective. A well-designed system uses RAG to retrieve the right 5,000 tokens from a 10M-token codebase and places them into a 128K context window alongside conversation history and tool outputs.

If you are building a coding assistant, an internal developer tool, or a code-aware agent, here is a reference stack that covers the key decisions.

Parsing and chunking

Use tree-sitter with the appropriate grammar for each language. The tree-sitter-python, tree-sitter-typescript, and tree-sitter-go grammars are mature and battle-tested. Extract function definitions (function_definition node type in Python) and class definitions as primary chunk types. For functions longer than your embedding model's context limit, split at the statement level within the function body.

# Pseudo-code: AST chunk extraction with tree-sitter (v0.21+ API)
import tree_sitter_python as tspython
from tree_sitter import Language, Parser

PY_LANGUAGE = Language(tspython.language())
parser = Parser(PY_LANGUAGE)

TARGET_TYPES = {"function_definition", "class_definition"}

def walk_tree(node, source_code: str, file_path: str, chunks: list):
    """Recursively walk the AST to catch nested definitions
    (methods inside classes, functions inside functions, etc.)"""
    if node.type in TARGET_TYPES:
        chunk_text = source_code[node.start_byte:node.end_byte]
        chunks.append({
            "text": chunk_text,
            "file": file_path,
            "start_line": node.start_point[0],
            "end_line": node.end_point[0],
            "type": node.type
        })
        # For class_definition, continue recursing to capture methods.
        # For function_definition, stop — we want the whole function,
        # not its nested helpers as separate chunks.
        if node.type == "class_definition":
            for child in node.children:
                walk_tree(child, source_code, file_path, chunks)
    else:
        for child in node.children:
            walk_tree(child, source_code, file_path, chunks)

def extract_chunks(source_code: str, file_path: str) -> list[dict]:
    tree = parser.parse(source_code.encode())
    chunks = []
    walk_tree(tree.root_node, source_code, file_path, chunks)
    return chunks

Embedding model selection

For a production code assistant, the embedding model choice matters significantly. Three strong options:

Vector store

For a single-developer or small-team tool: pgvector in a local Postgres instance is often sufficient. For a service handling multiple users: Qdrant is a strong choice — it supports both dense and sparse vectors in a single collection, enabling native hybrid search without maintaining two separate stores.

BM25 index

For pure BM25, Tantivy (the Rust-based full-text search library behind Qdrant's sparse vector support) or Elasticsearch's BM25 are the standard options. For a simpler deployment, the Python rank_bm25 library is adequate for corpora under ~50,000 chunks.

The prompt injection template

Placement and formatting of retrieved context matters. A template that works well in practice:

You are a coding assistant for this codebase.

## Relevant context from the codebase:

### [payments/gateway.py · lines 42–87]
```python
{chunk_1_text}
```

### [payments/exceptions.py · lines 1–24]
```python
{chunk_2_text}
```

### [payments/models.py · lines 88–112]
```python
{chunk_3_text}
```

## Current task:
{user_request}

Including file path and line numbers in each chunk header gives the model two useful signals: the module structure of the codebase, and the ability to reference specific locations in its response. These headers cost very few tokens but significantly improve the quality of generated code that needs to import from or reference the retrieved files.