Everyone is talking about Kimi K3, but if you jump straight into the technical report, you’ll quickly realize it’s standing on years of research – just like any breakthrough is! If you want to understand the work put into it by the Kimi team, here’s the reading order I’d recommend. Linear Transformers Are Secretly Fast Weight Programmers This is the foundation. The paper provides one of the most influential interpretations of linear attention, showing that many linear attention mechanisms can be viewed as fast weight programmers. Instead of thinking of attention purely as pairwise token interactions, it frames linear attention as a system that continuously updates an associative memory. Without understanding this perspective, it’s difficult to appreciate why modern linear-attention architectures have become competitive again. Gated DeltaNet (arXiv:2412.06464) Once you’re comfortable with linear attention, move on to Gated DeltaNet. This paper introduces the gated delta update mechanism, improving how state is updated over long sequences. Rather than using fixed update rules, the model learns when and how much information should be written into memory. Many of the ideas that later appear in Moonshot AI’s work build directly on these state-update concepts. Kimi Linear / Kimi Delta Attention (KDA) This is where Moonshot AI introduces the architecture that ultimately becomes the backbone of Kimi K3. Kimi Linear presents Kimi Delta Attention (KDA), a hybrid linear-attention architecture designed to combine the efficiency of linear attention with competitive or better performance than full attention across short contexts, long contexts, and reinforcement learning settings. Understanding KDA is essential because Kimi K3 is built on it. LatentMoE (arXiv:2601.18089) → Stable LatentMoE Kimi K3 isn’t just about attention. It also significantly advances the Mixture-of-Experts (MoE) design. Start with LatentMoE, which introduces a latent-space routing formulation that enables much higher sparsity while maintaining strong model quality. Then study Stable LatentMoE, Moonshot AI’s evolution of those ideas, which is used in Kimi K3 to efficiently scale sparse expert routing. In K3, Stable LatentMoE activates 16 out of 896 routed experts per token, contributing to its reported scaling efficiency improvements. Attention Residuals (arXiv:2603.15031) Residual connections have remained largely unchanged since Transformers were introduced. Attention Residuals asks a simple question: What if instead of naively squishing all these residuals together, we let the model decide how it wanted to use the residual network? (thanks to this person for framing the question correctly: mine version was little wrongly framed) Kimi K3 adopts this mechanism to improve information flow across model depth while keeping the approach practical for large-scale training. Follow the Kimi model evolution Finally, read the Kimi model reports in order: Kimi K1.5 – reinforcement learning scaling and reasoning. Kimi K2 – continued scaling of the architecture and training pipeline. Kimi K2.5 – multimodal and agentic improvements. Kimi K3 – integrates Kimi Delta Attention, Attention Residuals, Stable LatentMoE, refined training recipes, infrastructure advances, and large-scale reinforcement learning into a single frontier model. Reading them sequentially makes it much easier to see how the architecture evolved instead of viewing K3 as an isolated release. The biggest takeaway is that Kimi K3 didn’t appear overnight. It’s the result of multiple research threads converging: • Linear attention foundations • Better recurrent state updates • A stronger linear-attention architecture (KDA) • More scalable sparse MoE routing • Improved residual connections • Successive generations of Kimi models that integrated and refined these ideas If you’re planning to study the Kimi K3 technical report in depth, this reading path will give you the context needed to understand why the architectural choices were made—not just what they are. submitted by /u/East-Muffin-6472
Originally posted by u/East-Muffin-6472 on r/ArtificialInteligence