The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

Edge0 targets the weight half of the memory wall: a 35B Mixture-of-Experts model such as Qwen3.6-35B-A3B still occupies 19.5 GB at int4 even though only about 3B parameters activate per token. Instead of keeping experts resident, Edge0 memory-maps them from SSD and uses a trained per-layer prerouter to predict the next layer’s expert selection one token ahead. That prediction becomes the routing itself, allowing staged SSD reads to overlap decoding without dropped experts or fallback loads. A frozen int4 model is paired with an unmerged recovery LoRA, trained on the student path with prerouter routing, supervised fine-tuning, and on-policy distillation; keeping the adapter separate avoids the degradation caused by merging and requantizing it. On a 24 GB Apple-silicon machine using MLX, the Qwen-based 35B tier reaches 20.4 tok/s with only 2.9 GiB of peak active memory, versus 3.9 tok/s and 18.2 GiB for a fully resident baseline. On a storage-stressed 16 GB system, prerouting improves K=4 decode from 3.5 to 6.4 tok/s, an 82% gain. Across AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro, and IFBench, the served 35B model averages 79.2 versus 83.2 for its fp16 teacher. The framework also supports DeepSeek-style sigmoid-group routing and an 8B Ling hybrid tier, with open checkpoints and adapters.