How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

This controlled study tests whether on-demand AI assistance improves immediate performance while weakening later skill development. In a three-phase logic-puzzle experiment, 124 participants solved ordering problems before, during, and after a 20-minute AI-access phase; the simulated assistant was perfectly correct, revealing one object’s location per request. Lower assistance costs increased reliance: participants in the Low-cost AI condition made 6.67 requests on average, compared with 3.33 in the High-cost AI condition. Among participants who never used AI, reward rate rose from 2.03 to 3.86 correct objects per minute, a 90.2% increase, whereas Phase 3 performance was lower for AI users, at 3.42 versus 3.86 for non-users. A Bayesian latent ability model, estimated with MCMC, separated initial ability, post-assistance ability, and individual skill change. Its key result is that preserved independent reasoning, measured as solo share, predicted learning more strongly than request frequency: the solo-share coefficient was 0.125, while an alternative AI-usage coefficient was essentially zero. A one-minute increase in independent reasoning during the 20-minute AI phase corresponded to 1.5% higher predicted Phase 3 accuracy and 2.5% lower predicted response time. The findings suggest that AI assistance is not inherently detrimental, but becomes associated with weaker transfer when it substitutes for reasoning—a design concern relevant to systems such as ChatGPT, especially in education and training.