A symbol chunk needs its enclosing scope attached

The index is rebuilt on syntax boundaries. Parsing a file with tree-sitter, a parser library that ships grammars for most languages in common use, produces an abstract syntax tree — the AST, a tree whose nodes are the declarations, statements and expressions the language's own grammar recognises — and cutting on the boundaries of those nodes is what makes a chunk a whole function rather than a span of characters. So: a tree-sitter parse of every Python file, one chunk per function and per method, each chunk carrying its file path and its byte range, and nothing is severed any more. The next ticket says eviction is showing up in profiles as 8% of request latency on the hot shard and asks for it to be batched. Retrieval returns ShardedLRUCache._evict as a complete, syntactically valid unit, which is exactly what the previous index could not do, and the patch that comes back introduces a KeyError in the read path about once every 40,000 requests.

A patch that is correct in isolation#

Here is the class, with the part that matters to the ticket and the part the chunk did not include.

class ShardedLRUCache:
    """Size-bounded cache with a secondary index from tag to keys.

    Invariant: a key must be removed from `self._index` before it is
    removed from `self._map`. Readers take no lock — they resolve a tag
    to a key set through the index and then dereference each key in the
    map — so a key present in the index and absent from the map raises
    KeyError inside a request. The reverse ordering is safe: a key in
    the map that is no longer in the index is simply unreachable and
    will be collected on the next eviction pass.
    """

    def __init__(self, max_bytes: int) -> None:
        self._order: OrderedDict[str, Entry] = OrderedDict()
        self._map: dict[str, bytes] = {}
        self._index: dict[str, set[str]] = defaultdict(set)
        self._bytes = 0

    def _evict(self) -> None:
        key, entry = self._order.popitem(last=False)
        self._index[entry.tag].discard(key)
        del self._map[key]
        self._bytes -= entry.size

The chunk that was retrieved is the four lines of _evict and its signature. Read on its own, the ordering of the two deletions looks arbitrary, because within the method nothing depends on it. The agent batches the eviction the way anyone would, grouping the map deletions into one pass and collapsing the index updates by tag so that each tag's key set is rewritten once instead of once per key.

    def _evict(self, count: int = 32) -> None:
        victims = [self._order.popitem(last=False) for _ in range(count)]

        for key, _entry in victims:
            del self._map[key]

        by_tag: dict[str, set[str]] = defaultdict(set)
        for key, entry in victims:
            by_tag[entry.tag].add(key)
        for tag, keys in by_tag.items():
            self._index[tag] -= keys

        self._bytes -= sum(entry.size for _key, entry in victims)

This is a competent patch. It does what the ticket asked, it is faster, it type-checks, and the cache's own unit tests pass because they are single-threaded and only assert on the state after _evict returns. It also inverts the ordering the class docstring specifies as an invariant, meaning a condition the class promises will hold at every moment another thread can observe it, so for the microseconds between the map deletions and the index rewrite there are up to 32 keys visible in the index and absent from the map, and any reader resolving those tags in that window dereferences a key that is gone.

The information the agent needed was fifteen lines above the method, in the enclosing class. The chunk boundary is now syntactically correct and the retrieval unit is still incomplete, because a method is not a self-contained thing — it is a thing that means something relative to the scope it lives in.

The header a chunk cannot infer#

The minimal repair is to stop storing a method as though it were a top-level function and start storing it with its lineage attached. Every method chunk carries the class signature and the class docstring above it, marked so the model can tell what is context and what is the unit under discussion.

# billing/cache.py
class ShardedLRUCache:
    """Size-bounded cache with a secondary index from tag to keys.

    Invariant: a key must be removed from `self._index` before it is
    removed from `self._map`. Readers take no lock ...
    """

    # ---- chunk body ----
    def _evict(self) -> None:
        key, entry = self._order.popitem(last=False)
        self._index[entry.tag].discard(key)
        del self._map[key]
        self._bytes -= entry.size

The same move applies one level up. A chunk from a Go file is stored with the package clause and the import block prepended, because a call to errors.Is means something different depending on whether errors is the standard library or an internal wrapper, and the import block is the only place that resolves it. A TypeScript chunk carries the module's import statements for the same reason. A Java method carries its package declaration, the class signature, and the class-level annotations, since @Transactional on the class changes what every method in it does. The general shape is that a chunk stores its own text plus the smallest set of enclosing declarations required to interpret that text, which is normally the file header and the chain of enclosing scopes.

That header is also where the retrieval metadata comes from, and it is worth storing separately from the embedded text: the file path, the fully qualified symbol name, the kind of the node, the byte range, and the parent chain.

{
  "chunk_id": "billing/cache.py::ShardedLRUCache._evict",
  "path": "billing/cache.py",
  "symbol": "ShardedLRUCache._evict",
  "kind": "method",
  "parents": ["ShardedLRUCache"],
  "start_byte": 2814,
  "end_byte": 3092,
  "language": "python",
  "header_bytes": 612
}

A lexical query for _evict now hits a chunk that contains the name. A query for the class name hits every method in it. The byte range means a retrieved chunk can be expanded to the full file cheaply when the agent decides it needs more, and it means a chunk can be invalidated precisely when the file changes. None of this is available from a character offset into a text blob.

There is a choice of granularity hiding underneath this, and it is not obvious which way to go. One chunk per method retrieves precisely and loses the relationship between methods, so an agent that needs to know how put and _evict interact has to retrieve both and hope the ranker agreed. One chunk per class retrieves the interactions and blunts the precision, since a 400-line class embeds as an average of eleven concerns and matches nothing sharply. The arrangement that behaves best in practice is to index both, marking the class-level chunk as a coarse unit and the method chunks as fine ones, then deduplicating at retrieval time so that a result set never contains a method and the class chunk that already includes it. That costs roughly 1.9 times the storage of methods alone and removes the need to guess which granularity a given task wants.

The cost of the header is duplication. A class with twelve methods and a 40-line docstring stores that docstring twelve times, and on a repository with heavy class-level documentation the index grows by something between 15% and 40% in stored characters. Embedding cost rises with it. The trade is usually worth taking, because the alternative is patches that violate invariants stated in the text you chose not to store.

What the parser costs you#

The measurement in the cAST work, which compares AST-structured chunking against naive splitting on code retrieval, comes out in favour of structural chunking, and it is easy to read that as a settled matter. Structure-aware chunking is the right default. It is not free, and the three prices are worth being explicit about.

A parser per language is the first. Tree-sitter grammars exist for the languages a monorepo is likely to contain, but each one has its own node type names, so the code that says "this node is a function definition, this one is a class" is written once per language. A repository with Python, TypeScript, Go, SQL migrations, Terraform, protobuf definitions and shell scripts needs seven of those mappings, three of which will be written badly by whoever draws the short straw, and all of which need updating when a grammar changes node names.

Files that will not parse are the second. This is not rare. An editing session hands the indexer a file with an unclosed brace. Templating languages produce files that are not valid in either the host language or the template language — a Jinja-flavoured .py, an ERB-flavoured .rb, a .vue single-file component with three grammars in it. Generated code sometimes uses constructs the grammar version installed does not support. A parse failure has to fail to something rather than dropping the file, and a silent drop is the most common way a repository ends up with a hole in its index that nobody notices for months.

Uneven chunk sizes are the third, and they are structural rather than accidental. Measured on one mid-sized Python service of 180,000 lines, symbol chunking produced a median chunk of 340 characters, a 90th percentile of 2,100, and a maximum of 61,000 — a range of nearly three orders of magnitude. A fixed window produces a distribution with no variance at all, which is genuinely useful: every chunk costs the same to embed, retrieval returns a predictable number of tokens, and the context budget is arithmetic rather than a lottery.

A 550-line generated function#

Take the concrete pathology. A 600-line file, machine-generated from a protobuf schema, consisting of a header and one 550-line function that populates a lookup table. Symbol chunking produces exactly one chunk of about 61,000 characters, roughly 16,000 tokens, which exceeds the 8,192-token input limit of a common embedding model. The client truncates it, silently, and stores a vector that represents the first third of a table of constants. Nothing errors. The index reports one more chunk indexed.

Whether that is acceptable depends entirely on whether you can see it. The behaviour itself is fine — nobody needs semantic retrieval over generated lookup tables, and a truncated vector for such a file costs almost nothing in practice. The problem is a pipeline that does this without telling you, because the same silence covers the case where a hand-written 900-line service class gets truncated at the same threshold and its last eleven methods become unretrievable. The repair is a size cap on chunks with a defined overflow behaviour: a symbol body over some limit is split at statement boundaries into pieces, each carrying the same header and a part index, and the split is counted in a metric you look at. Generated files are better handled by detecting them and indexing only their symbol names, since their bodies are derivable from a schema that is itself in the repository.

Which one to ship on a repository you have not seen#

The defensible answer is symbol chunking with a per-file fallback: parse the file, and if the parse fails or the language has no grammar configured, chunk that file by fixed window and record which path it took. That gives structural boundaries where they are available and coverage where they are not, and it makes the degradation visible rather than turning it into missing rows.

Three things are worth measuring in the first week, and the third is the one people skip. Parse success rate by language and by directory tells you where the fallback is carrying the load; a language sitting at 71% success usually means one bad node-type mapping rather than 29% broken files. The chunk size distribution, specifically the count of chunks above the embedding model's input limit, tells you how much of the index is truncated. And retrieval quality on a labelled set of real tasks tells you whether any of it mattered, which is a harder thing to build honestly than it sounds and is the subject of a later chapter in this topic.

What symbol chunking does not do is worth stating plainly, because the improvement is real enough to be mistaken for a solution. Chunks are still ranked by similarity between a task description and a piece of code, so the bias from the first chapter — prose matches prose — is reduced rather than removed, and a well-documented irrelevant function still outranks a bare relevant one. More importantly, every chunk is still an island. The _evict chunk now knows about its class; it does not know which three modules call ShardedLRUCache.put, or that the tag index is also read by a background compaction job in a package that shares no vocabulary with the ticket. Those relationships exist in the repository as import and call edges, and the next chapter is about traversing them, because the file a change must touch is frequently one the words will never reach.