NITP: Next Implicit Token Prediction for LLM Pre-training

NITP, or Next Implicit Token Prediction, is a new pre-training objective for large language models that augments standard next-token prediction with dense supervision in hidden-state space. The paper shows that ordinary next-token prediction provides sparse, one-hot gradients that leave most latent degrees of freedom under-constrained, causing last-layer representations to become anisotropic and low-rank during training. NITP fixes this by asking the model to predict an “implicit token,” defined as the stop-gradient shallow-layer representation of the next token, and aligning the final hidden state to it with a cosine-similarity loss plus the usual cross-entropy. The key design choice is temporal shift: predicting the next token’s semantic representation works, while same-position alignment collapses and hurts performance. Theoretical analysis shows that NITP adds positive curvature to the angular subspace of the loss Hessian, mitigating the semantic null space left by next-token prediction. Empirically, the method scales across Dense and MoE models from 0.5B to 9B parameters, including DeepSeek-V2-style MoE backbones, with roughly 2% extra training FLOPs and zero inference overhead. On the 9B MoE model, NITP improves MMLU-Pro by 5.71 points, C3 by 6.36 points, and CommonsenseQA by 4.26 points, while also raising frozen-representation quality on MTEB from 39.24 to 41.56 across 25 tasks. The authors position NITP as a compute-efficient way to make pre-training shape not just token accuracy, but the geometry and transferability of internal representations.