The Mask Is Not the Model: Auditing Prefix Invariance in Attention, State-Space, and Hybrid Sequence Models

The paper introduces prefix invariance as the defining causality condition for autoregressive sequence models: changing a future token must leave every earlier layer representation unchanged. Its audit uses per-layer forward hooks, a single-token suffix perturbation, and two forward passes with caching disabled; it requires no gradients, labels, training, or accelerator, and reports the first offending layer. This is broader than inspecting an attention mask, which can miss leakage caused by scans, aggregation, normalization, or unmasked state-space and recurrent mixers such as Mamba and RWKV. Across 192 injected-fault trials spanning attention, state-space, recurrent, and hybrid architectures, the audit exactly localized all 192 faults, while static mask inspection detected 0 of 192. The study also found a real implementation defect in the PyTorch chunked-scan paths for Zamba2-1.2B and NVIDIA’s Nemotron-H-8B: an inter-chunk recurrence reduced over the output-chunk axis instead of the input-chunk axis, allowing future information into earlier positions at chunk boundaries of 256 and 128, respectively. A reference-axis patch removed the leak. The failure was invisible when sequences stayed below the relevant chunk size, motivating a rule that audit lengths must exceed architectural chunk, window, and kernel parameters. The authors also require a positive-control liveness test on the same loaded checkpoint, after inert Falcon-H1 loads produced apparently clean but unauditable outputs; Microsoft’s Phi-4-mini-flash-reasoning was likewise among models blocked by unavailable custom kernels.