FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution

FreeToken is an edge-native serving system for sparse mixture-of-experts models, designed for personal machines rather than datacenter clusters. It targets models such as DeepSeek-V4-Flash, GLM-5.2, Qwen3.6-35B-A3B, and OpenAI’s GPT-OSS, while supporting real agent workloads including Claude Code and tool-using applications. Its core contribution is bandwidth-adaptive execution: a shared GPU LRU expert cache captures token-to-token routing locality, while the q⋆ policy divides cache misses between PCIe transfers and direct CPU execution according to measured host and interconnect bandwidth. During prefill, full-layer double buffering overlaps expert loading with GPU computation; semantic checkpoints at thinking and tool-call boundaries let recurrent state and prefixes survive agentic context edits, avoiding redundant recomputation. FreeToken also resizes the expert cache dynamically as VRAM and KV-cache demands change, and uses NVIDIA NVFP4 or native model quantization without modifying model computation. On an RTX 5090, it reaches up to 83 tok/s on Qwen3.6-35B-A3B and achieves as much as 2.3× the decode throughput of edge baselines including llama.cpp, Ollama, and KTransformers. Its worst-case time to first token remains below 44 s, compared with baseline stalls above 150 s. The system serves a 35B model at 39.3 tok/s on an 8 GB RTX 4060 laptop, and runs the 753B GLM-5.2 at workstation scale, demonstrating that bandwidth-aware software can bring frontier open-weight models to consumer hardware.