SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

SimFoundry, from NVIDIA with collaborators at Stanford, Georgia Tech, and UT Austin, is a modular real-to-sim pipeline that turns a single RGB video into an interactive, sim-ready digital twin and then expands it into affordance-preserving object, scene, and task cousins for robot policy learning and evaluation. The system decomposes reconstruction into extraction, mesh generation, pose alignment, articulation inference, physics annotation, and PyBullet stabilization, while using foundation models such as DepthAnything3, SAM3, FoundationPose, Gemini-Pro-3, Hunyuan3D 2.1, and TRELLIS.2. Across 12 reconstructed tabletop scenes, it reaches zero-shot F1 scores of 0.81–0.92 and improves to 0.93–0.99 with only 3 minutes of per-object tuning; compared with SAM3D, it lowers reconstruction error and raises geometric fidelity. For real-to-sim evaluation across 7 manipulation tasks and 5 policy architectures, SimFoundry’s simulated rankings track real-world success with mean Pearson correlation 0.911 and mean MMRV 0.018, exceeding the PolaRiS baseline by more than 0.59 Pearson points. For sim-to-real training, object cousins, scene cousins, and task cousins improve average real-world task success by 17%, 21%, and 40%, respectively, while multi-task cousin data raises generalist policy performance by up to 31% in simulation and 18% in the real world, including 29% success on held-out tasks. The automatic background pipeline also outperforms manual alignment, with PSNR 15.29 versus 12.91 and NCC 0.749 versus 0.549.