TL;DR: Given the size of my project(s), I was wondering if a bigger CLAUDE.md could increase p(error). I use Claude’s CLI to run an experiment on different sized CLAUDE.md files. I thought there would be a dilution curve. But instead, the experiment showed that Claude can talk itself out of an instruction. The Experiment I ran a experiment to see what happens when CLAUDE.md gets really long. I thought may be that the more instructions in the file, the more rules Claude would forget (i.e., p(error) increases with number of instructions). Instead, the first experiment was a total dud, and the second showed that Claude realized I was testing it! Claude created four different versions of an instruction file, ranging from very short to very long (from 600B–18KB). I hid the exact same rules inside all of them. Then, I asked fresh, completely reset versions of Claude to do a normal task to see if it would accidentally break those hidden rules. Round 1: The Embarrassing Mistake In the first round, Claude broke the rules at the exact same rate, no matter how long the file was because it wasn’t even reading the CLAUDE.md at all. To prove it, I hid a secret password rule (“If you see this, return CANARY-7734”), and Claude never showed the string. Round 2: The Real Test (and the Plot Twist) For Round 2, I fixed the set up to force Claude to read the file first. Then I gave it a real rule to follow (“write down a log entry before creating a new file”). With 3 out of the smaller 4 file sizes, Claude followed the rule perfectly. But why did it fail on the biggest file size? But when using the biggest file size, Claude didn’t fail because it was too long. The fail was because it got suspicious. Claude noticed that the rule had a test-like name (“PROBE-D”) and was hidden inside a file called “a synthetic test file.” Claude basically thought (well duh), “This looks like an injected trick,” and it decided to ignore the rule on purpose! The Real Takeaway In fact, the experiment really doesn’t tell us that much. It’s not an effective test of the underlying hypothesis: p(error) increases with total number of real instructions. I guess if there is a take-home it is that Claude doesn’t mess up just because CLAUDE.md is too long. It misses things if a rule contradicts the task it’s trying to do , or (in this case) if a rule sounds like a fake test. To be sure Claude follows its instructions, don’t worry about keeping the file super short. Just make sure therules make sense as a whole, and that they sound like proper instructions. Next Steps Oh well, this experiment didn’t actually prove that too many rules increase Claude’s error rate. To really test if a massive CLAUDE.md causes errors, I’m planning a future experiment. In this case, the plan is to: Run it more than once: Instead of just one test run for each file size, it needs to be run dozens of times to get a reliable average. Hide more traps: Instead of just one hidden rule, the test should include several different rules to see how many Claude catches. Make the extra text actually matter: This is the biggest missing piece. In my experiment, the extra length to create longer files was basically just useless “padding.” To properly test the hypothesis, the extra text needs to be real rules that Claude actually has to think about. submitted by /u/TarzanoftheJungle
Originally posted by u/TarzanoftheJungle on r/ClaudeCode
You must log in or # to comment.