HyperDrone: Simulation-to-Reality Transfer at Millions of Frames per Second
AuthorsJonas Eschmann, Giuseppe Loianno
AffiliationsUniversity of California, Berkeley
HyperDrone aims to make realistic drone simulation fast enough to train vision-based flight policies at massive scale, then transfer them to a real quadrotor.
Key results
Frames per second in the 20-object, RGB, high-fidelity rendering test.
Times faster than the fastest existing quadrotor simulators, reported as more than 20×.
Procedurally generated indoor scenes available in the dataset.
Parameters in the student vision model.
Hours on an NVIDIA RTX 5090.
What the paper found
HyperDrone combines GPU-based quadrotor dynamics with HyperLux, a ray-tracing renderer that uses NVIDIA OptiX, Vulkan, Apple Metal, or WebGPU to produce RGB, depth, segmentation, surface-normal, and optical-flow images, with options for physically based rendering, reflections, anti-aliasing, and motion blur. In a 20-object test, it reached 11,489,513 frames per second on an NVIDIA RTX 5090 using Vulkan; on the same hardware class, the paper reports more than 20× the throughput of the fastest existing quadrotor simulators. To test simulation-to-reality transfer, the team trained a 0.73M-parameter vision model on synthetic ProcTHOR scenes to estimate yaw from current and target frames, then ran it onboard a real Crazyflie; the RAPTOR policy handled low-level flight control. Training took 4 h on an RTX 5090, and the policy successfully tracked target frames in real-world tests. The approach offers high-throughput synthetic training across 10K procedurally generated scenes, though the demonstration still relies on external position measurements and does not yet address rolling-shutter cameras or autonomous collision avoidance.
Original abstract
We present HyperDrone, an extremely fast quadrotor simulator focused on high-performance batch rendering that is more than 20x faster than existing state-of-the-art batch quadrotor simulators. HyperDrone combines highly optimized multirotor dynamics with HyperLux, a new ray-tracing engine that supports Nvidia OptiX, Vulkan, Apple Metal and WebGPU backends that takes full advantage of modern ray-tracing hardware, capable of running both components entirely on the GPU. HyperLux renders RGB, depth, semantic/instance segmentation, surface normals and optical flow observations at configurable levels of fidelity, including reflections, realistic lighting, motion blur, anti-aliasing, and Physically Based Rendering (PBR). Even in the highest-fidelity settings, HyperLux achieves millions of Frames Per Second (FPS) on consumer-grade GPUs while rendering realistic 3D scenes. We demonstrate the merits of HyperDrone by training a vision-based yaw-control policy purely in simulation and transferring it to a real small-scale quadrotor (Crazyflie) with a camera, running completely onboard.
Read the original paperMore in Robotics
Browse all 59 papers →DiscoDemo: Discovering Efficient and Diverse Robot Demonstrations for Imitation Learning
Minho Park, Kinam Kim, Donghu Kim, Byungkun Lee, Dongyoon Hwang, Yongjae Shin, Junha Hyung, Hojoon Lee, Jaegul Choo
DiscoDemo trains robots to find varied successful ways to complete tasks, then turns those behaviors into better imitation-learning data.
HoloBoxing: A Simple Recipe for Humanoid Boxing with 7 Minutes of Single-GPU Training
Xihan Ma, Bo Zhang, Boxiong Li, Yuhan Bao, Ziyue Liu, Yucheng Wang
A pretrained humanoid controller helps a compact planner learn simulated boxing in about seven minutes of additional single-GPU training.
Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts
Antonia Bronars, Younghyo Park, Kushal Kedia, Pulkit Agrawal
A robot learns to insert, mesh, and thread parts by feeling contact through its own joints, without vision or tactile sensors.