Memory went from 52% of what an AI accelerator costs to build at the start of 2024 to 63% by the end of 2025. Samsung put that on a slide at Hot Chips 2026 and sourced it to Epoch AI. Everything else in the package, processor included, comes to about a third of the bill now. Inside one of these packages there’s more memory silicon than processor silicon, a ratio Micron puts above eight to one. Arithmetic has been getting roughly three times faster every two years, and the memory feeding it hasn’t managed to double in the same window, so more of the accelerator’s life goes on waiting for numbers to show up. Once memory is two thirds of the cost, cutting the data a chip has to move buys you more than shrinking the logic onto a newer manufacturing process, and that reverses how the industry has ranked its priorities for fifty years. Memory makers have started designing parts for one customer at a time, which they didn’t bother with while the product was a standard commodity. DRAM spot price per gigabyte climbed around seven times over while the bits shipped barely moved, so most of what gets called a shortage is price doing the work. Wafer capacity has sat flat for over a decade and a new plant takes more than two years to stand up, which let the three companies still making this stuff book record revenue without adding much supply. There’s a wrinkle in reading that 63% share, since memory’s slice grew eleven points at the same time its own price multiplied several times over, meaning the rest of the package got more expensive too. Samsung put multiply units inside the memory chip itself so the data never has to travel to the processor, and on real silicon that came out at about three times the tokens per second on Llama 3.1 8B. A company called Oxmiq is pitching stacked flash for the weights a model only reads now and then, which buys something like fourteen times the capacity if you can live with a bit over half the bandwidth. Memory and HBM chapter: https://allaboutchips.com/#memory Free primer on the chips behind AI: https://allaboutchips.com/ submitted by /u/Summit-Star001
Originally posted by u/Summit-Star001 on r/ArtificialInteligence