So this was not even something I originally thought was realistically possible on my pc lol. I have a 4070 Ti 12gb, 32gb ram, 7800X3D and a pretty fast Samsung NVMe and I had basically assumed anything around 120B was completely out of my weight class unless I built some stupid expensive workstation with like 128gb+ ram or multiple GPUs. Well apparently not 😂😂 I had Codex messing around with this idea I had that I was calling CRANE, basically trying to find a way to run GPT-OSS 120B without needing to actually keep the entire fucking model in ram/vram at once. The model is like 59gb in MXFP4 so obviously my 32gb ram + 12gb vram isnt fitting that normally lol. The first normal llama.cpp attempt basically just annihilated Windows commit memory and my watchdog killed it before the whole pc turned into mashed potatoes. It got into like the 94%+ commit range almost immediately. So instead of trying to load it normally we basically started abusing the fact that GPT-OSS 120B is MoE. The general idea ended up being: shared/static model stuff stays resident hot experts stay on the GPU cold experts stay on the NVMe when an expert is needed it gets streamed into fixed buffers then over time it figures out what experts keep getting used and holds those in a persistent GPU cache instead of rereading them from the SSD every token At first it was hilariously slow but it ACTUALLY WORKED. First successful 120B generation was around: 2.63 tok/s generation 3ish tok/s prompt and it streamed like 18gb worth of expert data for only 18 evaluated tokens LOL So basically the SSD was getting its ass beat because it was pulling roughly a gigabyte of expert data per token. But once we knew it actually worked Codex started progressively caching hot experts. It went something like: 2.6 tok/s then 3.7 then with 4 hot slots around 7 tok/s 12 slots got around 9.2 tok/s sustained then once the cache was warm another agent turn hit like 13.8 tok/s 14 slots got around 15.1 tok/s At this point I told Codex fuck it push 20 😂😂😂 And somehow the bastard did it. With 16 adaptive GPU hot expert positions and a more aggressive top-1 approximation mode it hit: 21.16 tok/s generation and around 60 tok/s prompt processing on GPT-OSS 120B. On a fucking 4070 Ti. 😭😭😭😭 Important asterisk because I know somebody is gonna point it out: the 21 tok/s mode uses a top-1 approximation so this isnt me claiming untouched fully canonical GPT-OSS 120B inference magically does 21 tok/s on a 4070 Ti. Im keeping a slower fidelity mode too so I can actually compare how much the approximation changes output/model quality. But the actual 59gb GPT-OSS 120B checkpoint is being run locally and even before the aggressive approximation it was already usable once the hot expert cache started working. The funniest thing is I originally asked about making this whole custom runtime and the first estimate was basically like 4-7 months and hundreds of engineering hours 😂😂😂 Then I basically told Codex stop trying to reinvent everything and just go find existing open source shit we can smash together until something works. About 2 hours later we had a 120B model generating locally. Classic caveman engineering: find good rock smash rock into other rock benchmark rock throw away bad rock smash again And somehow the final rock runs a 120B LLM at 21 tokens/sec 💀 Im building a local AI sandbox/agent app called JANUS too, so now the next stupid idea is plugging this into that and letting the 120B model run autonomous simulations and tool use locally. Still genuinely cannot believe this runs on my pc lol. submitted by /u/JayB_Official
Originally posted by u/JayB_Official on r/ArtificialInteligence