Randomness in large language models: What researchers need to know (and report)

“Randomness in Large Language Models” argues that LLM-generated variables should be treated as draws from a distribution, not fixed measurements. Variation persists through deliberate sampling, silent provider updates, floating-point rounding affected by batching and hardware, model splitting, and mixture-of-experts routing. Setting temperature to zero removes deliberate sampling but not infrastructure or model changes; this distinction matters for proprietary APIs from OpenAI and Anthropic, where researchers cannot preserve the serving stack. The authors examine sentiment classification of 10-K filings from 100 S&P 500 firms, repeating the same prompt 200 times with OpenAI’s GPT-5.6 Luna, DeepSeek V4-Pro, and Google’s Gemma 4 26B A4B. With GPT-5.6, roughly 5% of regression t-statistics crossed a 90% significance threshold across repeated runs, showing how unstable LLM annotations can alter empirical conclusions. Consistent with prior evidence, nominally deterministic settings can produce accuracy differences of up to 15 percentage points, while GPT-4 performance on prime-number classification reportedly fell from 84% to 51% after a silent update. In the experiment, DeepSeek remained variable at temperature zero, whereas locally deployed Gemma became fully deterministic. The paper recommends archiving exact model versions, prompts, settings, timestamps, hardware and software environments, token and cost records, and every raw output. Open-weight systems such as Gemma, DeepSeek, and Meta’s Llama offer greater control than hosted models, but reproducibility still requires preserving the complete execution stack.