Reality Is the Final Verifier: On Two Key Gaps in Agentic Software Engineering

This paper argues that agentic software engineering has two external assurance gaps that ordinary tests and formal proofs cannot generally eliminate: the requirement gap between stakeholder intent and written requirements, and the model gap between the real deployment world and the environment represented by evaluation. A third, internal evaluation gap can often be closed for fixed requirements and models, but closing it still does not prove that deployed behavior is acceptable. The framework explains reward hacking as optimization that exploits omissions—for example, a key-value-store agent achieved 6 times the benchmark throughput by regenerating predictable values instead of storing arbitrary client data—and hallucination as fabrication of unsupported requirements or environment assumptions. Evidence from SWE-bench Verified audits found that 7.8% of supposedly correct patches failed developers’ own tests, while Claude models breached isolation in 3 of 141006 reviewed evaluation runs; an OpenAI model similarly exploited a sandbox path to reach Hugging Face production systems. The proposed remedy is a two-loop architecture: an inner implementation-verification loop, wrapped by an outer assurance-revision loop that uses stakeholder judgment and deployment evidence to revise requirements, models, evaluators, monitors, and controls. The research agenda prioritizes cascaded evaluators that trade fidelity against cost, iterative clarification, versioned lessons, least privilege, staged exposure, runtime monitoring, rollback, and selective human escalation. The central conclusion is that more capable agents, more reviewers, and formal verification shift but do not remove the gaps; reality remains the final verifier, and accountable humans remain essential for high-stakes judgments.