Cross-Primitive Consistency in Jev
AuthorsXu Liu, Guikun Chen, Kanzhi Wu, Wenguan Wang
AffiliationsZhejiang University · vivo
PrimaCons measures when LLM decisions change across equivalent query formats, and CPRC helps make those decisions more consistent.
Key results
Covers complement, label-hierarchy, abstention-alignment, and label-description consistency.
Cross-primitive probability discrepancies predict decision errors.
Reduction reported for CPRC under leave-one-domain-out evaluation.
Relative improvement in raw-label accuracy with CPRC.
Latency overhead introduced by CPRC.
What the paper found
This paper introduces PRIMACONS, a four-axis benchmark for testing whether Jev decision models give consistent outputs when the same decision is expressed through different querying primitives. Using HaluEval, PKU-SafeRLHF, AbstentionBench, and AG News, it evaluates complement, label-hierarchy, abstention-alignment, and label-description consistency across JevK5-2B, JevK5-4B, and JevK5-9B. The models show systematic contradictions: finer label categories increase divergence, the choice of primitive affects the direction of disagreement, and adding descriptive text can worsen it. Notably, cross-primitive probability discrepancies predict decision errors with a ROC-AUC of 0.91. To address these failures, the paper proposes the Cross-Primitive Residual Calibrator, or CPRC, which keeps the backbone frozen and uses a lightweight residual network trained to improve accuracy while aligning predictions. In leave-one-domain-out evaluation, CPRC reduces inconsistency by 58.57% and improves raw-label accuracy by 2.97% relative, with only 0.75% additional latency. The results suggest that agreement across equivalent query formats can serve as a practical reliability signal for structured decision models.
Original abstract
Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated. This work introduces PrimaCons, a four-axis benchmark for assessing cross-primitive consistency while preserving the semantics of the query. Evaluations across three Jev sizes reveal consistency violations across all four axes. Our analysis shows that these violations follow systematic patterns and that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91. To mitigate these inconsistencies, we propose the Cross-Primitive Residual Calibrator (CPRC), which attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency. Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%., demonstrating its effectiveness in both in-domain and out-of-domain settings. Our findings establish cross-primitive consistency as a key dimension for evaluating Jev and provide a practical approach to improving the reliability of its decisions. Code and benchmark is available at https://github.com/67L1/PrimaCons-CRPR.
Read the original paperMore in Large Language Models
Browse all 99 papers →All Quiet on the Neural Front: Language Models Do Not Report Injected Pain States
Emilio Ferrara
Models’ internal signals can reveal an injected pain-like state and change their behavior, even when the models themselves do not report feeling it.
Knowing When to Teach: Learning from Teachable Prefixes in Multilingual Reasoning Distillation
Jaewon Cho, Minseo Yoon, Jinyoung Park, Jinwoo Seo, Jaewon Chu, Alice Oh, Hyunwoo J. Kim
The paper teaches multilingual reasoning models only on parts of their own reasoning where a teacher can still reliably guide them.
On the Generalization Limitation of On-Policy Distillation for Post-RL Consolidation
Yichen Wang, Yufa Zhou, Siqi Zhu, Haozheng Luo, Jingzhi Bao, Zhichao Liu, Zijun Wei
Distilling a model’s skills can preserve benchmark performance while losing its ability to generalize, and this paper explores ways to prevent that.