Mobile AI Factory: Building Specialized Language Models for Your Mobile Device
AuthorsEvan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota
AffiliationsPaul G. Allen School of Computer Science and Engineering, University of Washington
Mobile AI Factory automatically builds and tunes a small language model for a specific task and phone, aiming to deliver strong performance with less compute.
Key results
Maximum score advantage over five-shot Claude Sonnet 5 and DeepSeek v4 Flash.
Model variants in the biomedical entity-extraction search comparison.
Fold less GPU time than exhaustively training all variants.
Claude Opus 5.5’s specification-based selections missed the app requirements.
Claude Opus 5.5’s specification-based selections missed the app requirements.
What the paper found
Mobile AI Factory automates the creation of a task-specific, quantized small language model for a particular phone. An orchestrator powered by Claude Sonnet 5 chooses whether to tune LoRA settings, mine real examples, or generate targeted synthetic data from a Qwen3.6-35B-A3B teacher; deterministic rules roll back accuracy regressions and move the search from smaller to larger models. Crucially, the system profiles each phone’s latency, memory, and energy across llama.cpp and MNN, then selects a feasible model, quantization, runtime, and CPU or GPU backend. The paper situates this approach alongside mobile runtimes including Meta’s ExecuTorch, but focuses on measured deployments using llama.cpp and MNN. Across ten tasks, the factory’s models beat five-shot Claude Sonnet 5 and DeepSeek v4 Flash on five unseen-test tasks, with gains up to 0.342, and matched them on three more. On biomedical entity extraction, its search came within 0.03 of exhaustively training all 27 model variants while using 19-fold less GPU time. Phone measurements also proved essential: Claude Opus 5.5, given hardware and model specifications but no measurements, selected configurations that missed requirements in 25% of easy and moderate cases and 60% of hard cases. The result is a system that treats fine-tuning, quantization, and real-device profiling as one search problem, rather than assuming the most accurate model will fit or run well on every phone.
Original abstract
Small language models (SLMs) running on mobile devices can keep user data private and work without connectivity. But they perform well only when fine-tuned for the target task and sized for the target device, a process that today requires repeated expert tuning. We present {\it Mobile AI Factory}, an agentic system that builds a fine-tuned, quantized SLM for a given task and phone. Given a task specification and a target phone, an orchestrator LLM chooses whether to tune hyperparameters, mine real data, or generate synthetic examples, while fixed rules roll back changes that hurt accuracy. The factory searches from small to large models across base models and quantizations, adapts its accuracy goal as it searches, and uses real measurements of the target phone across inference engines and backends. We evaluate Mobile AI Factory on ten tasks and four Android phones. On an unseen testset, its on-device models outperform two five-shot frontier models on five tasks, by up to 0.34, and match them on three more. Its search comes within 0.03 of exhaustively training all 27 model variants on an entity extraction task, while using 19$\times$ less GPU time. Phone measurements also matter: the best configuration changes across phones. Without phone measurements, Claude Opus 5.5, given the full hardware and model specifications, misses the app's requirements in 25\% of easy and moderate cases and in 60\% of hard ones. Code is available at: https://github.com/evanly-gh/SLM_Factory.
Read the original paperMore in Efficient AI
Browse all 79 papers →Latent Feedback Is a Compute and Data Amplifier
Benhao Huang, Chaitanya Dwivedi, Xuezhe Ma, Alexander Moreno, Hector Liu
Latent feedback may let language models get more value from each training token and FLOP than simply repeating the data.
MACaQ: Fast, Model Agnostic Calibrated Quantization for Low-Precision Weights and Activations
Lorenz K. Muller, Philippe Bich, Chiara Boretti, Hyun-Min Chang, Jiawei Zhuang, Lukas Cavigelli
MACaQ aims to make large neural networks cheaper to run by quantizing their weights and activations quickly, with little added inference overhead.
FLAMIA: FLAsh MIcroscaling Attention for Fast and Causally Safe Training on Blackwell
Fedor Velikonivtsev, Mikhail Koltakov, Denis Kuznedelev
FLAMIA makes Transformer attention training faster on Blackwell GPUs by carefully combining low-precision computation with safeguards for causal correctness.