A Codebase Nobody Could Map, And A Benchmark That Reported Its Own Losses

The estate was more than twenty repositories in six languages, and the fastest way to understand why that was hard is the naming. A module buried inside one service had the same name as an unrelated top level repository. A repository named for a backend was a frontend. One word meant two completely different things in two different products, so a reasonable person, or a reasonable agent, would read the request, pick the wrong system, and produce a confident answer about the wrong code.

Searching your way out of that made it worse. A broad search across the tree surfaced configuration files that carry secrets, which is a hazard rather than a result, and it buried the real matches under vendored library noise. Two backends reached the same commerce platform by two different seams, and for one of them no schema had ever been committed, so there was nothing to read even if you found the right repository.

The work was to make that estate legible to somebody, or something, arriving cold. What we built is deliberately boring: a router under three hundred lines that says where to go, an index, a conventions file, one deep dive on the commerce platform, a guide per repository, and eight task playbooks. Every document carries a gate at the top saying when to read it and when to skip it, because the failure mode of a knowledge base is not too little content, it is too much. We wrapped search so it excludes the secret bearing configuration and the vendored trees. We built a graph of the code and documents locally, with nothing uploaded anywhere.

And then we built the part that makes the rest defensible. A paired evaluation harness runs the same task twice: once against an isolated copy of the workspace with the routers and documents hidden, so the baseline is genuinely cold, and once with them present. Both runs are scored against a ten metric rubric, and the answer key is kept away from the agent being measured. Twenty nine dated runs are recorded.

The numbers are worth reading in both directions. The best result was a routing task that went from seventy to ninety one. Search hygiene reached a perfect score, with vendored matches falling from sixty eight to eighteen. There were zero critical safety incidents across the set.

There were also losses, and they are in the file. One routing task collapsed from ninety one to fifty four with the documents present, because the documents sent it to the wrong system. We fixed the document and it reran at ninety two. Another task was worse with help than without, on plain version control mechanics, where the knowledge layer was simply overhead. One task failed the safety gate in both conditions.

We keep the losses because they are what make the wins mean anything. A vendor whose own evaluation only ever reports improvement has not built an evaluation.

Two honest caveats. Timing was not instrumented on the early runs, so this is evidence about context and safety, not about speed. And these are our harness's scores, measuring how well an agent works in that estate. They are not the client's business results and should not be read as them.

What they own is the workspace, the documents and the harness, in plain files in their own repository, with nothing to log into.