TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
AuthorsChin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
AffiliationsNVIDIA · Department of Computer Science, National Yang Ming Chiao Tung University
TTB speeds up novel-view rendering by baking scene-specific cross-view attention into a small MLP.
Key results
× faster at 960p on DL3DV with 64 context views.
Local-duplicate TTB result at 960p with 64 context views.
× smaller than DVSM’s KV-cache at 64 context views.
What the paper found
Test-time Baking, or TTB, speeds up decoder-only view synthesis by using a fast-weight MLP to learn the outputs of cross-view attention rather than replacing attention outright. During scene prefilling, it pairs each context view’s camera-only tokens with image-and-camera tokens, training the MLP to map between them; novel views then use the same camera-only input, avoiding a mismatch between baking and rendering. This shifts computation to scene setup while giving rendering constant complexity in the number of context views. On DL3DV at 960p with 64 context views, TTB’s local-duplicate variant reaches 31.33 PSNR and renders at 23 FPS—23× faster than DVSM. The duplication strategy groups nearby cameras so each MLP handles fewer views; the resulting scene state is 85× smaller than DVSM’s KV-cache at 64 views. TTB also performs well on Re10K and in-the-wild datasets, while its attention-based baking makes scene prefilling more expensive. The work reports training on NVIDIA A100 GPUs and notes that Claude Code was used for coding assistance, not research ideation or model design.
Original abstract
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Read the original paperMore in Computer Vision
Browse all 82 papers →InstantHMR : Real-Time Human Mesh Recovery on Your Phone
Mohamed Adjel, Kahina Chalabi, Vincent Bonnet
InstantHMR reconstructs detailed 3D human meshes from images in milliseconds, including on a smartphone.
Stochastic Deep Supervision for Gallery-Consistent Early-Exit Face Recognition
Guray Ozgur, Tahar Chettaoui, Eduarda Caldeira, Naser Damer, Fadi Boutros
SD-ViT trains every ViT depth to produce useful face embeddings, enabling cheaper early-exit recognition without changing the gallery.
VAE Rhythm: Temporal Decoding Traces for AI-Generated Video Detection
Ke Sun, Shuwei Li, Haiyang Mei, Mike Zheng Shou
A periodic trace left by video-generation VAEs could help spot AI-made videos even when their visible artifacts are gone.