Scroll to bottom for tldr In July, JetBrains reran the headline claims of two token-saving tools on real agent workloads. Caveman claimed 65% and measured 8.5%. RTK claimed 60–90% and ended up slightly more expensive than using nothing. It looked like a pattern of over claiming numbers, so I benchmarked 5 token saving tools with conditions closer to how agents actually use them my setup was : 48 Django questions drawn from SWE-bench Five question types, selected before running anything Same agent, prompt, repository commit and tool access Fresh index for every tool One no-tools baseline 261 runs on this harness 43 of the 48 questions finished in all six arms. The run hit an API usage cap near the end and lost the last five from every arm equally, so the paired comparisons are over those 43. Disclosure: I work on repowise. The harness, raw data, preregistration files and invalidated runs are all public Codex : gpt-5.6-sol Nobody saved 60%. The best result was about 32% of the agent’s output tokens, and once you correct for testing five tools at once, three of those five reductions hold up and two are marginal. CodeGraph is a real second at 24.4%, so the correct reading is that more than one tool here works. Serena is the odd row. It writes less than the bare agent while calling tools 42% more often, which makes it busier rather than leaner. Indexing is also part of the tradeoff. Repowise saved the most tokens here but took the longest to index, because it builds several additional intelligence layers in the same pass. For a straightforward call graph, CodeGraph is 22x faster. And 366.8s is the version with prose generation switched off; a default init on this repo is 1,058s. Why there is no Claude Code table here I ran the same questions, servers and indexes under Claude Code with Sonnet 5, and then again with Opus. Those tables are on the benchmarks page, because under Claude Code most of these tools were barely called at all. code-review-graph was never called once across 15 questions. Graphify was called three times, Serena four. Nothing was different about the servers, the questions or the indexes between the two harnesses, and Codex called every tool on every question. The likely explanation is harness behavior. Claude Code loads MCP schemas on demand, so the agent has to go looking before it can call anything, and frequently never does. Codex mounts them up front I plan to rerun Claude with hooks enforcing tool adoption to see the savings when the tools actually get used Quality There was no meaningful quality winner, including repowise. A blind judge scored every tool in the field, mine included, a fraction below the bare agent, in a range of 0.04 to 0.25 points on a 10 point scale. None of those gaps is distinguishable from zero, and all of them are smaller than the 0.69 points the same benchmark moves when it is rerun unchanged. Deterministic retrieval benchmark Token counts still depend on an LLM deciding what to write, so I also ran a deterministic benchmark using ContextBench Each task has a known list of files touched by the real fix. The score simply measures whether a tool retrieves those files. So there was no LLM judge. Coverage on its rewards whoever serves highest files, which is why precision and files served sit next to it. get_answer finds the most and hands back about 19 files to do it. code-review-graph finds the least and is the most precise thing in the table, 0.240 from 5.4 files, so if you are paying per token that row reads better than its coverage suggests. Graphify serves 34.5 files for 0.546 This required 748 index builds and roughly 78 hours of indexing for 1,129 graded instance/tool pairs. Every tool indexed every repository independently at the task’s original base commit, with no shared cache. Two mistakes I nearly published I nearly published a cost table showing code-review-graph as 43% cheaper than the baseline, in the same run where Claude Code never called it once. The reason was prompt-cache warming. Whichever arm ran first paid the full price, and later arms reused the cache That is why the tables report output tokens rather than API cost. There is a larger version of this mistake, which is measuring one retrieved payload instead of the complete agent session. Loading one commit’s context through repowise takes 393 tokens against 13,984 for reading the changed files, so 35.6x, and that is the easy number this whole category tends to publish. The full-session figure is 31.6% on Codex and 15.9% on Claude Code. Agents re-read, backtrack and re-plan, so a compression that looks enormous on a single payload nets out much smaller across a session. Full methodology, raw data and reproducibility files: https://github.com/repowise-dev/repowise/blob/main/docs/BENCHMARKS.md I’m happy to add other tools if there are ones people think are worth testing. The harness is public, so you can also rerun or challenge the results directly. TL;DR: I benchmarked five codebase tools on 48 Django tasks under Codex. None came close to the commonly advertised 60–90% savings. Repowise saved the most output tokens at 31.6%, CodeGraph was close behind at 24.4%, and the rest landed between 6% and 15%. The same questions under Claude Code produced a harness result rather than a tool result. Most of the field was barely called at all, one tool never once, despite nothing changing on the tool side, so those tables live on the benchmarks page. Answer quality differences were smaller than the evaluator’s own noise, and every tool including repowise scored slightly below the bare agent. On a separate deterministic retrieval benchmark, repowise found 87.6% of the files touched by the real fixes from about 19 files served, while code-review-graph found 44.5% from 5.4. Measure full agent sessions, and always report the harness, indexing cost and cache effects alongside token savings. submitted by /u/Obvious_Gap_5768
Originally posted by u/Obvious_Gap_5768 on r/ClaudeCode