Thought without systematicity? Evaluating reasoning models on rule induction tasks

The paper tests whether reasoning models exhibit systematicity: the ability to preserve task-solving competence when symbols, feature bindings, example order, or constituent arrangements change without altering the underlying rule. It constructs structurally equivalent variants of four synthetic rule-induction families—grammar-based instruction-learning, symbolic Raven’s progressive matrices, program induction over integer sequences, and Boolean category learning—and evaluates OpenAI’s GPT-5.5, Anthropic’s Claude Opus 4.7, and Google’s Gemini 3.1 models. Each task is expanded into K=5 isomorphic variations, with each variation judged by majority vote over N=5 sampled attempts. The central result is a systematicity gap: models often solve at least one variant but fail on equivalent variants, especially in Raven’s matrices and integer-sequence program induction. For Claude Opus 4.7, 91.67% of program-induction variation sets had at least one solved variant, but only 50.00% had all variants solved, a 41.67% gap. Boolean category learning was the exception, with almost all models solving nearly every variation. Increasing reasoning effort from low to high did not consistently improve systematicity, and Gemini’s zero-temperature evaluation showed that the gap persists even when sampling noise is largely removed. The findings challenge the assumption that success on a single reasoning benchmark reliably indicates a transferable underlying capability.