What I built skillassay — a Skill (and CLI) that measures what your CLAUDE.md and Skills actually cost in always-on context, and flags Skills that a compliant client is entitled to silently ignore. Free, Apache-2.0, runs offline, no API key. It walks CLAUDE.md including nested chains, .claude/skills/** , .claude/agents/** and .mcp.json , and separates what loads every session from what only loads on demand. Every finding carries a rule ID, a citation, and a token count. I fetched 1,029 published SKILL.md files from 11 public repos to validate it. 1,022 parsed (99.32%). 37 have hard spec violations — 26 with a name: that doesn’t match its parent directory, 8 with illegal characters or casing, 3 with a description over the 1,024-char limit. Another 58 have no detectable trigger clause telling the model when to load them. How Claude Code was used Claude Code wrote most of the implementation. I want to be specific about this because “built with Claude Code” usually means the easy 70%, and the remaining 30% is where the actual work was. The pattern that ended up mattering: treat everything it writes about measurement as unverified until it runs on real data. Concretely, three habits: Falsifiability tests. For every check that guards correctness, I wrote a test that plants a violation and asserts the check fires. This caught a check of mine that was completely dead: no-unfinished-code was scanning comment-stripped lines, while TODO markers live in comments. It passed cleanly and detected nothing, forever. A check that can never fail also passes — so passing tells you nothing unless you’ve proven it can fail. A sanitized publish tree. My working directory has CLAUDE.md , a dev log, and build notes I don’t want on GitHub. So there’s a build step that generates a clean ship/ directory with those excluded, runs a verification pass for forbidden patterns, and that’s the only thing that gets pushed or published. Genuinely recommend this if you build with Claude Code — the internal context files accumulate fast and you will eventually push one by accident. (I found mine had already committed a 94 KB tarball I never intended to publish.) Run it on real repos, not fixtures. More below. What I learned
- It will emit numbers that nothing computed. My first prototype had an –empirical flag printing things like “+2.1% task success” and “$1.42 saved.” Entirely plausible. No code produced them. I deleted the flag and wrote a lint rule that fails the build on fabricated-looking output. If you’re building anything that reports measurements with an LLM, build that check — you cannot catch this by reading the output, because the output looks correct.
- Green unit tests caught none of my real bugs. Every precision bug passed the fixture suite and was only visible on real data: a regex for detecting directory trees matched every code fence in every README a package-manager check using includes(‘bun’) matched the word “bundle” one version summed every context file on disk and reported 123,567 tokens for a repo whose real always-on cost was 21,086 — a 6x overstatement, because it counted files that never load in the same session Running on 30+ real repos found bugs at roughly 2 per repo. Nothing else I did came close in value per hour.
- Test the artifact you shipped, not your dev tree. After publishing I ran the published binary through 54 adversarial scenarios. Found that –json truncated silently when piped: Bash $ assay . --json > out.json # 70,320 bytes, valid $ assay . --json | cat | wc -c # 65,536 bytes — exactly the pipe buffer Node writes to a pipe asynchronously and process.exit() doesn’t flush it. So redirecting to a file worked while piping to jq got half a document — which is exactly how an agent consumes the tool. My dev tree never showed it because I’d only ever redirected to files.
- Token counts need their method attached. There is no offline Claude tokenizer, so every count is cl100k_base and the output labels it a proxy. It would have been easy to print a confident number. The label is the honest version. The honest limitation Median saving across 33 real repos was 0 tokens . 18 of 33 had nothing worth deleting. If your context is already lean it says so rather than inventing work. It was clearly more useful to people writing Skills than to ordinary app repos. Stating that up front seems better than having you install it and discover it. Current state: 150 tests, CI on Ubuntu/macOS/Windows × Node 20/22/24. –fix prints a diff to stdout and nothing else — no code path writes to your files. Try it npx skillassay . # whole repo npx skillassay path/to/SKILL.md # one Skill, while authoring Or install it as a Skill and just ask “audit this repo’s context cost” — it costs 98 always-on tokens and passes its own audit with zero findings. Repo, method, measured accuracy, known limits, and every false positive from my own calibration: https://github.com/rakib-nyc/skillassay submitted by /u/theawkwardbong
Originally posted by u/theawkwardbong on r/ClaudeCode