Original Reddit post

Original Article: https://x.com/p_nawrot/status/2089315591010079034 I’ve spent the last few years working on efficient attention and KV Cache Compression. I’ve read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I’ve learned a few things. One of them is definitely “how to make things look good, even when they aren’t.” I’m guilty too, but trying to get better every day.

  1. For single-hop retrieval, make sure there are no distractors and context is useless The three most cooperative settings for compression / sparsity are: Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text. Contaminated benchmarks from years ago for which models don’t even look at the context anymore. Few-shot in-context learning, where extra shots are useless and don’t improve the accuracy over 0-shot. With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn’t matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity.
  2. NEVER isolate your contribution Short context: Most of a dense model’s performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it’s neither relevant to nor the subject of this post. Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let’s go, don’t look back. Do the same with block size. Smaller blocks can give you finer granularity and more precision in retrieval, so keep their old block size and make yours smaller. Ignore the fact that things may get slower due to irregular memory accesses, etc. Those were historical decisions; respect them. 🤡 Write: “We used the authors’ recommended hyperparameters.”, then spend weeks tuning your method. The same trick works for speed. LLMs are pretty good at writing Triton now. Keep the baseline algos exactly as they were written in 2023, then ask an LLM for a custom Triton kernel for yours. Extra cleverness if, by using a more efficient implementation, you can hide that your method does more work. You’re just optimising your method, no? Prompts are the cherry on top. Move the question before the context so the model knows what to filter out, then present the result as lossless compression. Never share the prompts after tuning them. Don’t tune the baselines to reject your paper; tune yours until it’s accepted.
  3. Use aggregated metrics to hide areas where your method doesn’t work RULER has 13 tasks: 6 NIAH tasks satisfy the first point. 2 QA tasks use datasets from years ago. VT also has a lot of irrelevant context. To be clear: This isn’t a critique of RULER; imo it’s still incredibly useful. It’s just an example of potential improper use. Report only the aggregate; maybe, in the limitations section at the end, briefly mention that your method degrades on the NIAH-MK3, which actually stress-tests lossless compression.
  4. Enjoy saturated tasks Imagine evaluating on two tasks: The most recent math exam / olympiad from a week ago, which isn’t yet in the training data. A benchmark on which a recent family of open models—1B, 10B, and 100B—all scored 80%. On the former task, before compression gets a chance to do any damage, the 1B and 10B models already score 0%; the 100B model starts at 50%, and its performance drops monotonically as compression increases. On the latter, all model sizes tolerate substantial compression, and the 100B model tolerates more than the 1B and 10B models. Don’t ask whether the larger model is simply using its extra parameters and hidden-state capacity to absorb compression in a setting where those resources aren’t needed to solve harder questions. That definitely isn’t what’s happening. Extras AIME has 30 samples. You did 4 seeds. Your method scores 80, and the baseline scores 79—bold your 80 and say that it surpasses the baseline. Statistics doesn’t exist. Bonus points for your efficiency method surpassing the baseline and setting a new SOTA. 🤡🤡 Pick a baseline, optimise it with your method, and plot a beautiful quality–efficiency curve against the original implementation. Then stop. Don’t ask whether a simpler route—a smaller dense model, KV-cache quantisation or offloading, or a better system configuration—reaches a better operating point. Improving your baseline is basically the same as improving the frontier. submitted by /u/korec1234

Originally posted by u/korec1234 on r/ArtificialInteligence