Your inference stack is burning money on solved problems. A storage hierarchy would fix the part you’re paying for twice.

Original Reddit post

Hey everyone, so I spent the last few weeks going down the KV cache rabbit hole. One thing which is most of what makes LLM inference expensive is the storage and data movement problems that I think database engineers solved decades ago. IMO, prefill is basically a buffer pool rebuild that nobody bothered to cache. So I did this write up using LMCache as the concrete example (tiered storage, chunked I/O, connectors that survive engine churn). Included a worked cost example for a 70B model and the stuff that quietly kills your hit rate. Curious what people are seeing in production. ✌️ submitted by /u/tirtha_s

Originally posted by u/tirtha_s on r/ArtificialInteligence