DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

DocOps, from researchers at the Chinese Academy of Sciences and Baidu, is a verifiable benchmark for autonomous agents that manipulate native XLSX, DOCX, PPTX, and PDF files rather than merely reading them. Its taxonomy spans three operation families—content, format, and structure—and four difficulty levels: L1 atomic edits, L2 composite edits, L3 single-document workflows, and L4 cross-document workflows. The benchmark contains 210 tasks, each evaluated by an offline deterministic verifier using structural predicates, linguistic anchors, and preservation checks against the final artifact. Across models and harnesses including OpenAI’s GPT-5.5 with Codex, Anthropic’s Claude Sonnet 4.6 with Claude Code, and open-source systems such as DeepSeek-V4-Pro and Qwen3.6-27B, the best configuration reaches only a 0.671 pass rate. GPT-5.5’s average pass rate falls from 0.725 on L1 tasks to 0.237 on L4 tasks, showing that long-horizon state tracking and cross-document consistency remain major bottlenecks. Excel workflows are especially fragile because agents must preserve formulas, references, validation boundaries, and sheet state. The study identifies three dominant failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing that flattens native structures or breaks formulas and tables. The verifier agrees with human judgments in 95.31% of 128 audited decisions and detects 96.67% of 180 injected violations. Execution harnesses strongly affect results: programmable environments with file-system feedback outperform constrained document APIs, while skills help selectively, improving Qwen3.5-27B under Claude Code by 7.1 percentage points but producing inconsistent gains overall.