I came across a project recently that claims to run Kimi K3 (2.78T parameter MoE) on a regular CPU with as little as 8GB of RAM. At first I thought it was complete BS, but after reading through how it actually works, it’s surprisingly legit. The important thing is that it isn’t loading the entire 1.5TB checkpoint into memory. That’s where most of the confusion comes from. Instead, the engine does a few interesting things:
- It takes advantage of the MoE architecture where only 16 out of 896 experts are active for each token.
- It streams expert weights directly from an NVMe SSD instead of trying to keep everything in RAM.
- It keeps recently used experts in an LRU cache so it doesn’t have to reload them every time.
- It performs computation directly on compressed MXFP4 weights instead of expanding them first.
- The whole inference engine is written in portable C99, without PyTorch, CUDA, TensorRT or even BLAS. Obviously there’s a catch. It’s slow. On lower-memory systems you’re looking at seconds per token, so this isn’t replacing GPU inference anytime soon. I don’t think anyone is deploying production chatbots like this. But I also don’t think that’s the point. To me, this feels more like a systems engineering project than an AI project. Instead of asking “How much RAM do we need?” it asks “Do we actually need all of the model in memory at the same time?” That’s a pretty interesting way to look at the problem. I honestly think ideas like streaming, smarter caching and better memory management are going to become much more important as models keep getting bigger. Curious what people here think. Is this actually the direction inference engines are heading, or is it just a really cool proof of concept that won’t have much practical impact? Polished with AI. submitted by /u/porAssass
Originally posted by u/porAssass on r/ArtificialInteligence
You must log in or # to comment.