JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
AuthorsJianwen Sun, Chuanhao Li, Zizhen Li, Yukang Feng, Fanrui Zhang, Yifei Huang, Yu Dai, Kaipeng Zhang
Resources
This paper creates the first large-scale benchmark for whole-game code generation in a real game engine, revealing that today’s AI can often write code that compiles but still fails to produce working games.
Key results
Projects surviving deterministic Godot-based filtering from over 240,000 repositories
Manually verified projects used as JamBench
Remaining verified projects used as JamSet
Task 2 runtime pass rate on Small projects before dropping on Large
Task 2 runtime pass rate on Large projects
Approximate candidate repositories searched during dataset construction
What the paper found
JAMER Project Level Code Framework Dataset and Benchmark on Professional Game Engines introduces JamSet and JamBench, the first project-level game-code framework dataset and benchmark built on the Godot engine. The core technical contribution is a deterministic verification pipeline that filters over 240,000 open-source Game Jam repositories down to 8,133 verified 2D projects by checking file integrity, compilation, and 30-second runtime stability in Godot’s headless mode; 300 of these are manually verified for evaluation, while 7,833 become training data. The benchmark spans theme-driven from-scratch generation and multi-granularity code completion, scored with compilation pass rates plus two new metrics, Structural Completeness Score and Behavioral Alignment Score, to separate runnable shells from behaviorally faithful games. Across 9 frontier models, the paper reports a sharp scale cliff in Task 2: runtime pass rates drop from 80.4% on Small projects to 5.7% on Large ones, and Code Agents mainly improve compilation without meaningful gains in SCS or BAS. Fine-tuning Qwen3.5-27B on JamSet improves engineering behavior, including higher use of input abstractions, autoload state management, and scene transitions, showing that curated project-level game data can shift models toward human-like architecture rather than just syntactic repair.
Original abstract
Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional game engines remains largely unexplored due to the absence of large-scale datasets and deterministic evaluation methods. We present JamSet and JamBench, the first project-level game code framework dataset and benchmark built on a professional game engine. Our key insight is that Game Jam competitions, community events where developers build complete games under tight time constraints, yield thousands of open-source projects suitable for this purpose. Building on the Godot engine's text-based format and headless execution mode, we design a deterministic verification pipeline from file integrity to runtime behavior collection, distilling 8,133 verified projects from over 240,000 repositories. Of these, 300 manually verified projects form JamBench; the rest constitute JamSet. JamBench defines theme-driven generation and code completion tasks, evaluated through a pipeline combining compilation pass rates, Structural Completeness Score (SCS), and Behavioral Alignment Score (BAS). Evaluation of 9 frontier models reveals a capability cliff as project scale increases, with runtime pass rates dropping from 80.4% on small projects to 5.7% on large ones (Task2a). Code Agents improve compilation rates yet yield no gains in runtime behavioral quality, indicating that the bottleneck lies in architectural design rather than syntactic correctness. Experiments validate JamSet as effective training data. All data and code are publicly available.
Read the original paperMore in Code Generation
Browse all 43 papers →Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer
Md Shohel Arman, Igor Molybog
Better code documentation can faithfully reconstruct software, but surprisingly does not necessarily help AI coding agents fix real issues when the source code is already available.
Groupwise Agentic Grading and Advantage Redistribution for Code Agent RL
Jinhao Dong, Liang Zhao, Zihao Yue, Wenhan Ma, Linghao Zhang, Lei Li, Shicheng Li, Yifan Song, Bowen Ye, Fuli Luo
GAGAR helps code agents learn not only to pass tests, but to produce cleaner and more targeted implementations by redistributing RL credit according to agentic quality judgments.
Reinforcement Learning from Intermediate Renders for Image-to-Code Generation
Omri Kaduri, Kate Feingold, Phillip Isola, Tali Dekel
IR4RL improves image-to-code generation by rewarding models for making useful visual progress at every intermediate rendering step.