Belief-Trajectory Energy: Measuring the Path to a Prediction
AuthorsJiahao Ying, Wei Tang, Boxian Ai, Yaoning Wang, Haotian Chen, Wenhe Sun, Caijun Xu, Haozhan Cai, Changyi Xiao, Yixin Cao
Affiliations[
Belief-Trajectory Energy tracks how an LLM’s predictions change across layers to reveal information that its final answer alone can miss.
Key results
Highest reported Spearman correlation between middle-layer BTE and task difficulty.
Qwen-3.5-9B using token-level BTE statistics to distinguish human reviews and seven generators.
Fold increase in OLMo-2-7B middle-layer BTE during pretraining.
Decrease in middle-layer BTE after math fine-tuning.
What the paper found
Belief-Trajectory Energy, or BTE, measures how a language model revises its next-token predictions across Transformer layers. Using the logit lens, it reads intermediate states as probability distributions and applies Jensen–Shannon divergence to each layer transition; locally, this corresponds to predictive change under Fisher–Rao geometry. Unlike a final-output score, the resulting depth profile preserves the path of revisions. As a training-free difficulty signal, BTE’s Spearman correlation with task difficulty reaches 0.66 across benchmarks including MATH-500, ProofWriter, StepGame, and SpaRTUN, though the signal depends on the scoring model and its training exposure. In paper-disjoint review detection across eight conference-years, Qwen-3.5-9B achieves 0.998 macro-AUROC. For eight-way attribution among human and model-written reviews, adding token-level statistics raises accuracy to 95.6%. The study also finds that OLMo-2-7B’s middle-layer BTE grows 7-fold during pretraining, while math fine-tuning reduces it by 22% on MATH-500, showing that the measure reflects what the scorer has learned. The review experiments include outputs from OpenAI’s GPT-5.5, Google’s Gemini 3.1 Pro, and Anthropic’s Claude Opus 5, scored using models including Llama-3.1-8B and Qwen-3.5-9B.
Original abstract
Large language models (LLMs) progressively revise their predictions across Transformer layers, yet we typically observe only the final output, discarding the trajectory through which it is formed. We introduce Belief-Trajectory Energy(BTE), a model-grounded measure that characterizes an input through the layerwise predictive revisions it induces in a model. By mapping intermediate states into a shared predictive space, BTE provides a principled measure of belief change that can be summarized as either a scalar or a structured depth profile. Theoretically, we show that local BTE corresponds to predictive revision under the Fisher-Rao geometry, while the sequence of revisions captures information beyond the initial-to-final belief change. Empirically, scalar BTE provides a model-relative signal of difficulty across diverse reasoning tasks, while richer BTE representations support human-LLM review detection and fine-grained generator attribution, reaching up to $0.998$ macro-AUROC and $95.6\%$ eight-way attribution accuracy. Further analysis shows that BTE develops throughout pretraining and is selectively reshaped by targeted training, demonstrating that the resulting measurement reflects what the scoring model has learned. Together, our results establish belief trajectories as a principled model-grounded signal and suggest a broader perspective in which learned models can themselves serve as instruments for characterizing the data they process. More demonstrations can be found at https://yingjiahao14.github.io/BTE-web/.
Read the original paperMore in Large Language Models
Browse all 90 papers →Competing Learning in Hybrid Architectures
Xingyu Shen, Yingfa Chen, Ziqing Qiao, Chaojun Xiao, Xu Han, Zhiyuan Liu
A staged training approach helps hybrid language models balance recurrence and attention, improving language modeling and retrieval.
On the Tip of the Tongue: First-Token Cueing as a Behavioral Probe of Recall Failures in LLMs
Jaewon Sok, Jewon Yeom, Seonghyeon Park, Jeongjae Park, Hwiyeong Lee, Taesup Kim
A single correct opening token can help an LLM recover factual answers it seemed unable to recall.
Copy-Paste Language Models
Nathan Godey, Yoav Artzi
Copy-Paste Language Models add a low-overhead copying mechanism that improves language-model training and performance, especially in long-context and retrieval settings.