GOFish: Full Field-of-View Gaussian Splatting and Mesh Extraction
AuthorsAllen Tu, Meher Gitika Karumuri, Shifan Zhu, Hailing Wang, Yen-Liang Lin, David Held, Matthias Zwicker, Tom Goldstein
AffiliationsAmazon University of Maryland, College Park University of Massachusetts Amherst 4 5 Northeastern University Carnegie Mellon University https://3dgofish.github.io
GOFish turns ultra-wide fisheye images into real-time-renderable 3D scenes with collision meshes for simulation.
Key results
dB improvement over the best prior method on the 200° FIORD benchmark.
dB improvement over the best prior method on the 205° FullCircle benchmark.
dB improvement over PPISP on FIORD.
Score achieved with the optional Geometry Objective.
FPS on an NVIDIA A10G GPU.
Fold reduction from Calibrated Mip Filtering in the reported ablation.
What the paper found
GOFish tackles a key limitation of Gaussian Splatting: existing fisheye methods miss parts of very wide views, while Gaussian mesh-reconstruction methods generally require rectified, pinhole images. In a project involving Amazon, the system trains directly on calibrated fisheye images and combines a calibration-defined ray domain, complete Gaussian-to-tile admission, and per-pixel ray-distance sorting to render beyond 180° and recover peripheral geometry. Calibrated densification builds out the surface, while Calibrated Mip Filtering and Wide-Angle Appearance Modeling—an extension of PPISP—keep pixel blur and lens falloff out of the shared scene and mesh. On FIORD’s 200° views, GOFish improves PSNR by 2.28 dB over the best prior method; on FullCircle’s 205° views, the gain is 1.97 dB. Its wide-angle appearance model alone adds 3.17 dB over PPISP on FIORD. With the optional Geometry Objective, its FIORD mesh reaches an F-score of 0.249. On an NVIDIA A10G, the real-time renderer reaches 79 FPS, and calibrated filtering cuts the Gaussian count by as much as 5.4× in the reported ablation. The result is a single scene representation that supports full-field rendering, ray tracing, and mesh-based collision geometry for simulation.
Original abstract
Robots, autonomous vehicles, and consumer 360° cameras use fisheye lenses for wide fields of view (FoVs). Simulation-ready digital twins require photorealistic rendering across the full FoV and an aligned collision mesh. However, prior fisheye Gaussian Splatting methods only rasterize up to 180° and do not produce meshes, while Gaussian surface reconstruction methods assume pinhole training images. We present GOFish, the first 3D Gaussian Splatting surface reconstruction method to train directly on fisheye images, rasterize FoVs beyond 180°, and extract an aligned mesh from the same scene. Calibrated densification recovers the surface across the entire FoV, while Calibrated Mip Filtering and Wide-Angle Appearance Modeling keep pixel blur and lens falloff out of the scene and mesh. Across the 200° FIORD and 205° FullCircle benchmarks, GOFish improves PSNR over the best prior method by 2.28 and 1.97 dB, respectively. It also outperforms fisheye rendering baselines within their own FoVs and achieves the highest mesh F-score on FIORD and the main ScanNet++ set. Together, the Gaussian scene and mesh form a LiDAR-free, simulation-ready digital twin that renders in real time, ray traces the full FoV, and provides collision geometry.
Read the original paperMore in Computer Vision
Browse all 82 papers →InstantHMR : Real-Time Human Mesh Recovery on Your Phone
Mohamed Adjel, Kahina Chalabi, Vincent Bonnet
InstantHMR reconstructs detailed 3D human meshes from images in milliseconds, including on a smartphone.
Stochastic Deep Supervision for Gallery-Consistent Early-Exit Face Recognition
Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira, Naser Damer, Fadi Boutros
SD-ViT trains every ViT depth to produce useful face embeddings, enabling cheaper early-exit recognition without changing the gallery.
VAE Rhythm: Temporal Decoding Traces for AI-Generated Video Detection
Ke Sun, Shuwei Li, Haiyang Mei, Mike Zheng Shou
A periodic trace left by video-generation VAEs could help spot AI-made videos even when their visible artifacts are gone.