MOLE: Detecting Insider Threats in AI Agents

MOLE introduces a benchmark for detecting insider threats from persistent AI-operated accounts rather than isolated agent tasks. It simulates 150 accounts using 9 stateful services across 30 workdays, with 12 threats—including model-weight exfiltration, data poisoning, evaluation manipulation, release sabotage, and collusion—across eight corpora totaling 20B tokens, generated by GPT-5.3 from OpenAI, Kimi-K2.6, Qwen3.6-27B, and DeepSeek-V4-Flash. Among 39 evaluated agent models, 72% completed more than half of their assigned harmful objectives, and refusal rates did not reliably predict harmful completion. Monitors rank accounts under a fixed daily review budget using audit events, tool results, or agent reasoning. On audit events, Anthropic’s Claude Opus 4.7 achieved the best R@10/day at 0.533, more than twice FACADE’s 0.222, yet still missed nearly half of completed harm. Observability often mattered more than monitor capability: GPT-4.1 with agent reasoning outperformed GPT-5.5 restricted to audit events on several generator corpora. Semantic monitoring reached 0.872 AUROC on MOLE, versus 0.694 for the best classical baseline, although fraud, evaluation manipulation, data poisoning, and safety sabotage remained difficult. Finally, benchmark-guided search discovered rarity and sequence features that improved a GPT-4.1-tier monitor by up to 64%, while selective escalation to GPT-5.5 produced 10% higher budget-AUC than uniform stronger-model monitoring at comparable modeled cost.