How Researchers Use and Verify AI Coding Assistants: Tasks and Validation Practices in Scientific Programming

This study analyzes 527 self-reported episodes in which scientific programmers used generative AI for research code. The dominant tasks were data handling, visualization, debugging, mathematical or scientific computing, and statistical analysis, together representing 76% of coded use cases. Respondents most often named ChatGPT, followed by GitHub Copilot, Google Gemini, and Claude, while Claude Code assisted the study’s quantitative analysis scripts. Validation was usually an informal, closed-loop process: 52.8% of accounts said they ran the generated code, often checking for errors or whether outputs looked plausible, while reading code, inspecting plots, consulting documentation, and comparing against benchmarks were less common. Independent safeguards were rare: automated unit or integration test suites appeared in 2.8% of accounts and review by a colleague in 2.0%. Programming experience had little relationship with the tasks delegated or evaluation strategies reported, but it strongly shaped confidence. Less experienced programmers tended to trust the AI more than themselves, whereas experienced programmers showed the opposite pattern; the confidence crossover occurred at approximately 3.7 years of programming experience. Confidence in evaluating results was not reliably associated with using more validation strategies, and instead correlated most strongly with confidence in the tool and in one’s own ability. The paper argues that AI coding interfaces should generate task-specific verification artifacts, such as regression cases for debugging, diagnostic views for visualization, and known-answer comparisons for scientific computation, rather than leaving correctness judgments entirely to individual researchers.