Yes you Heard that right no API, no PSRAM, no Clickbait just pure LLM model Running on 512KB SRAM the Model is Roughly 5.2 Million Parameter MoE With 16 Experts quantized to INT4 the Engine Basically Streams the Experts from the Flash to the SRAM and only Runs One Expert per Token Using around Only 81kb leaving 215kb for KV Cache and Improvements for Next versions i Used 6 Layers, 4 Heads and 128 embedding tokens Very small i know but Still Improving the Capacity The full Model Quantized Weights around 3.1mb (the bottleneck why i can’t just increase the size of the model) and the TPS (Tokens per Second) isn’t Bad at all infact it’s really good around 5 TPS on an ESP32 Dev kit V1 i also added a Math Harness so it can solve simple equation as the model is too small to solve it on it’s own and added Attention Sink to make the Context Window more bareable to use for a model that is running only using 81kb of memory the responses are pretty good for it’s size here is the github : https://github.com/ahmedbarakat207/espllm sooo check it out if you want :p https://preview.redd.it/k2x5654u0aih1.png?width=640&format=png&auto=webp&s=1bbc700a93aa4a01f8e17b9103613ec0ff8f94d1 submitted by /u/Similar_Wealth_1850
Originally posted by u/Similar_Wealth_1850 on r/ArtificialInteligence
You must log in or # to comment.