I’ve spent the last year convinced the industry is solving the wrong problem. We don’t need bigger models. We need smarter deployment. I would love to name my model “Iris Ai” Iris runs 8 specialized LLMs (Triage, Router, Control, Math, Code, Reasoning, General, Vision) on a single consumer machine with one hard constraint: M_active ≤ 1 at any execution timestamp. The peak memory footprint is bounded by the single largest active model, not the aggregate cluster. How routing works: - Phase 1: Zero-cost regex scanner catches unambiguous inputs (AST patterns, math operands, file paths) and routes directly — sub-millisecond, no neural inference - Phase 2: A small triage model (T=0.2, Top-P=0.1) handles ambiguous inputs and emits a single structured routing token from a closed vocabulary On model swaps, we run an explicit 3-phase flush: free KV cache → gc.collect() + cuda.empty_cache() / mlx.metal.clear_cache() → load new GGUF. Skipping the Metal cache flush on Apple Silicon causes the new model load to fight retained Metal allocations Benchmarks on our Medium tier (66B Total, ≤14B active, 16GB RAM): - 92.0% on HumanEval (pass@1, greedy decoding) Large tier (193B Total, ≤32B active, 24GB VRAM): - 95.0% HumanEval, 94.0% MATH Also A 40B math specialist beats a 200B generalist on math. A 14B code specialist hits 92% HumanEval on a MacBook Air. The proposition isn’t “bigger is better.” It’s that specialization + dynamic loading beats monolithic scaling on consumer hardware. Specialists were trained with LoRA (r=16/32, alpha=2r) targeting W_q/W_k/W_v/W_o, then delta matrices merged back into base weights — zero adapter overhead at inference. Math and Reasoning specialists used GRPO instead of RLHF, with reward signals for format compliance (<think> tags), compiler-verified syntax, and verbosity penalties. A few things worth noting for context: All benchmark scores are measured with dynamic output harnesses enabled (AST truncation repair, automatic import injection, LaTeX normalizer). The harness is part of the system — not a post-processing trick — but worth being transparent about. Current roadmap: - Overlapping model load with preceding token stream (eliminating perceived swap latency) - Online quantization hot-patches based on runtime memory pressure - Expanding GRPO training to Vision and Control specialists Here is the Source Code with Training Code and Data, The Routing Mechanism and and Benchmarks script: https://www.github.com/ahmedbarakat207/Iris-Ai MIT licensed. Everything runs on llama.cpp / GGUF. submitted by /u/Similar_Wealth_1850
Originally posted by u/Similar_Wealth_1850 on r/ArtificialInteligence
You must log in or # to comment.