Pruning and Distilling Mixture-of-Experts into Dense Language Models

This paper introduces the first systematic MoE-to-dense compression pipeline for language models, converting a trained Mixture-of-Experts teacher into a standard dense FFN through expert scoring, top-K selection, optional grouping/merging, block concatenation, and forward-KL distillation. The core novelty is DO-ACP, a D-optimal expert selector that maximizes the log-determinant of an importance-weighted Gram matrix, combining activation-weighted conditional probability with explicit redundancy penalty; greedy selection provides a 1−1/e approximation guarantee. On Qwen3-30B-A3B, the authors sweep 350 configurations across 7 scoring methods, 5 grouping methods, 2 down-projection scalings, and K∈{8,16,32,64,128}. Scoring dominates: DO-ACP reaches 43.41% average accuracy after 0.3B tokens of distillation, versus 37.17% for reverse-KL-style weak scoring baselines and 33.28% for a strong dense-to-dense pruning baseline from Qwen3-32B, a +10.1 point margin. Under matched parameter count, MoE-to-dense also trains 1.6× faster than dense-to-dense pruning, 73 s/step versus 116 s/step. After ∼4B tokens, DO-ACP scales to 58.10% average accuracy, beating dense-to-dense by +6.3 points and random FFN initialization by +12.7 points. The method generalizes to DeepSeek-V2-Lite and GPT-OSS-20B, where DO-ACP remains best and pure pruning at K=k is consistently optimal, showing that expert diversity, not averaging, is the key signal for converting MoE capacity into a compact dense student.