What I built densely - lossless context compression for agents. Big tool outputs (logs, JSON dumps) get re-encoded so Claude counts 5-15x fewer tokens, and the original is recoverable byte-exact (sha256-verified on every expand). Ships as an MCP server plus a PostToolUse hook that auto-compresses any large tool output before it lands in history. MIT. How Claude Code was used For basically all of it - someone on r/LLMDevs looked at the repo and wrote “The commit history is 100% claude.” Guilty. What actually made it work:
- A CLAUDE.md with hard behavioral rules (state assumptions before coding, surgical diffs only, every task phrased as a verifiable goal). Night-and-day difference versus no rules.
- Goals, not vibes: “compress+decompress must round-trip byte-exact on these 5 real files” instead of “make compression work”. Claude Code loops until the check passes.
- It ran the launch too: benchmarks, README, and replying on Reddit through my browser. The technical bit I’m proudest of: a commenter caught that our density claim was calibrated on OpenAI’s tokenizer, and on Claude it silently degraded ~3x. Anthropic’s vocabulary isn’t public - so Claude Code harvested a Claude-native carrier alphabet through the free count_tokens endpoint: ~45k API probes, batch exact-match plus bisection. Finding: Claude’s tokenizer has only ~1,035 single-token space-prefixed words in total (o200k has 68k). The resulting 1,024-word alphabet gives a guaranteed 10 bits per token and ships as the default. What I learned
- Hooks are underrated. PostToolUse intercepts a 74k-token tool output before it burns context; my own ledger shows 131k tokens saved in two days of normal use.
- count_tokens is a free oracle for Claude’s tokenizer. You can verify any token-count claim against the real counter in minutes.
- Real numbers, measured on Sonnet 5’s own counter (byte-exact round trip verified per row): a real macOS install.log slice went 104,328 -> 6,732 tokens (93.5% saved); live npm registry JSON 82.7%; a package-lock.json 62.5%; source code only 36-40% - and actively-edited code shouldn’t be compressed at all (the payload is unreadable by the model; it’s cold storage + server-side search/expand, not a summary). Repo: https://github.com/alibaizhanov/densely Longer write-up of the tokenizer reverse-engineering: https://densely.dev/blog/bpe-slack Honest limits are in the README. Happy to go deep on the hook setup or the alphabet harvesting. submitted by /u/No_Advertising2536
Originally posted by u/No_Advertising2536 on r/ClaudeCode
You must log in or # to comment.