I ran an independent benchmark of the token-saving tools that get recommended here (MCP code indexes, compressors, hooks, skills). Not affiliated with any of them, no sponsors.
- The noise is huge. Same six bug fixes, same pinned Claude Code build, no tools installed: across 7 identical runs the bill varied by up to 54.5%.
- Few savings held up. Across 15 tools × 3 agents × 2 repos, only 9 of 79 combinations beat that noise when tested again.
- First runs misled. 4 of 5 “wins” that looked real after one run fell back into the noise on a second run.
- Results don’t travel. With Claude Code, 7 tools held up on a JavaScript repo and zero on a Python repo.
- Method in one line: real upstream bugs, symptom-only prompts identical across runs, each tool installed per its docs, compared against the same agent running bare, measured as fresh input + 0.1 × cached + 6 × output.
- A 30% saving in one run isn’t a result. It’s a reason to run the test again. Charts and per-tool results: https://merlinhu1.github.io/token-optimization-research/ Raw data and method: https://github.com/merlinhu1/token-optimization-research If you think I set a tool up wrong, open an issue and I’ll re-run it Cheers! submitted by /u/Middle_Key8737
Originally posted by u/Middle_Key8737 on r/ClaudeCode
You must log in or # to comment.