LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routers

LLMRouter presents a unified infrastructure for choosing among heterogeneous language models by casting routing as a sequential decision process with five components: context encoders, model encoders, scoring functions, decision rules, and learning signals. This abstraction covers single-turn, multi-turn, agentic, and personalized routing, while its automated data engine runs every query across an 18-model pool—including Meta’s Llama, Mistral, Qwen, GPT-OSS, and DeepSeek-V3.1—then records task quality, token usage, and cost in a dense query–model matrix. The resulting xRouteBench contains 4,767 test instances spanning generic reasoning, memory, vision, video, time series, and personalized dialogue. The open-source library provides 17 built-in routers, supports pointwise, pairwise, and reinforcement-learning objectives, and exposes deployments through an OpenAI-compatible server, OpenClaw, and ComfyUI. Across the benchmark, learned routers deliver a 14.6% relative improvement over the strongest fixed-model baseline, while router rankings change sharply as cost constraints tighten, demonstrating that always selecting the largest model is inefficient. Personalization is effective but depends on how user context is represented: GMTRouter reaches 68.78 accuracy with a persona-conditioned DeepSeek-V3.1 judge, whereas PersonalizedRouter achieves 83.05 accuracy on held-out real-user preferences collected through Slack. The study shows that no router dominates every task or budget, and multi-turn decomposition does not consistently justify its added inference cost.