NTH

Cross-Primitive Consistency in Jev

AuthorsXu Liu, Guikun Chen, Kanzhi Wu, Wenguan Wang

AffiliationsZhejiang University · vivo

November 9, 2026 2 min read
Watch on YouTube
The one-line take

PrimaCons measures when LLM decisions change across equivalent query formats, and CPRC helps make those decisions more consistent.

Key results

4
PRIMACONS evaluation axes

Covers complement, label-hierarchy, abstention-alignment, and label-description consistency.

0.91
Error-prediction ROC-AUC

Cross-primitive probability discrepancies predict decision errors.

58.57%
LODO inconsistency reduction

Reduction reported for CPRC under leave-one-domain-out evaluation.

2.97%
LODO relative accuracy improvement

Relative improvement in raw-label accuracy with CPRC.

0.75%
Additional latency

Latency overhead introduced by CPRC.

What the paper found

This paper introduces PRIMACONS, a four-axis benchmark for testing whether Jev decision models give consistent outputs when the same decision is expressed through different querying primitives. Using HaluEval, PKU-SafeRLHF, AbstentionBench, and AG News, it evaluates complement, label-hierarchy, abstention-alignment, and label-description consistency across JevK5-2B, JevK5-4B, and JevK5-9B. The models show systematic contradictions: finer label categories increase divergence, the choice of primitive affects the direction of disagreement, and adding descriptive text can worsen it. Notably, cross-primitive probability discrepancies predict decision errors with a ROC-AUC of 0.91. To address these failures, the paper proposes the Cross-Primitive Residual Calibrator, or CPRC, which keeps the backbone frozen and uses a lightweight residual network trained to improve accuracy while aligning predictions. In leave-one-domain-out evaluation, CPRC reduces inconsistency by 58.57% and improves raw-label accuracy by 2.97% relative, with only 0.75% additional latency. The results suggest that agreement across equivalent query formats can serve as a practical reliability signal for structured decision models.

Original abstract

Jev enables efficient decision-making with large language models (LLMs) through typed querying primitives, allowing the same semantic state to be expressed in different formats, such as binary and multiple-choice queries. However, the consistency of decisions across primitives has not been systematically evaluated. This work introduces PrimaCons, a four-axis benchmark for assessing cross-primitive consistency while preserving the semantics of the query. Evaluations across three Jev sizes reveal consistency violations across all four axes. Our analysis shows that these violations follow systematic patterns and that cross-primitive discrepancies predict decision failure with a ROC-AUC of 0.91. To mitigate these inconsistencies, we propose the Cross-Primitive Residual Calibrator (CPRC), which attaches a lightweight residual network to a frozen backbone and jointly optimizes predictive accuracy and cross-primitive consistency. Under a leave-one-domain-out protocol, CPRC reduces cross-primitive inconsistency by 58.57% and improves accuracy by +2.97%., demonstrating its effectiveness in both in-domain and out-of-domain settings. Our findings establish cross-primitive consistency as a key dimension for evaluating Jev and provide a practical approach to improving the reliability of its decisions. Code and benchmark is available at https://github.com/67L1/PrimaCons-CRPR.

Read the original paper

More in Large Language Models

Browse all 99 papers →