VolFill

NeurIPS 2026

VolFill Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching

  • 1University of Massachusetts Amherst
  • 2Technical University of Crete
Paper arXiv Code Model BibTeX
VolFill · scene 1
Input RGB image for the selected scene
Input
One RGB image in, a complete 3D scene out, including surfaces the camera never saw. Every point cloud on this page is raw VolFill output; drag to look behind objects. As in the paper, surfaces seen in the input keep the photo's colour and hidden geometry recovered by VolFill is blue.

TL;DR

Pixel-aligned depth models only recover what the camera sees. VolFill treats the hidden part of the scene as a generation problem. It compresses a truncated unsigned distance grid with a hybrid sparse–dense 3D VAE. A diffusion transformer then samples the complete scene with flow matching, conditioned on a frozen geometry foundation model and anchored to the visible surfaces.

  • 3.4× lower Fréchet point-cloud distance than NOVA3R FPD 4.86 vs 16.50 · metric-scale evaluation
  • +14.3 occluded-surface coverage on SCRREAM APD0.02 55.53 vs 41.21 (NOVA3R)
  • +9.0 complete-scene F-score on SCRREAM FS0.02 54.83 vs 45.84 (best baseline)
  • 1.4s per scene on one RTX 4090 50 Euler steps with classifier-free guidance

01The problem

A camera only sees first hits

Depth models such as MoGe-2 and Depth Anything 3 predict one 3D point per pixel: where each ray first hits a surface. The result is a sharp 2.5D shell with nothing behind it. Amodal reconstruction asks for every surface in the camera frustum, visible or occluded. The pixels don't determine the hidden part, so VolFill doesn't regress it. It samples it from $P(V \mid \mathbf{I})$.

02Representation

Distance to the nearest surface, truncated

Each voxel of a $256^3$ grid stores its distance to the nearest surface, truncated at $\tau = 3$ voxels: $V(\mathbf{p}) = \min(\operatorname{dist}(\mathbf{p}, \mathcal{S}), \tau)$. The distance is unsigned because indoor geometry is open (single-sided walls, floors, furniture seen from one side), so inside and outside are undefined. Surfaces come back by thresholding $V < \epsilon$ into a point cloud, or by meshing with MeshUDF.

Try it

A real TUDF slice

A horizontal slice through a SCRREAM scene reconstructed by VolFill, seen from above with the camera at the bottom edge. Each pixel is one voxel. Drag $\tau$ to change the truncation band and $\epsilon$ to change the surface threshold.

Interactive slice loads here.

03Hybrid 3D VAE

Compress the grid into a small latent

A $256^3$ grid is too large to generate directly. A 3D VAE compresses it into a compact $16^3$ latent and decodes it back. The decoder is hybrid: it upsamples densely to $64^3$, keeps only the cells near a surface, and finishes at $256^3$ with sparse convolutions. Compared with a dense-only decoder, it is about 9× faster (0.07 s vs 0.63 s) and uses about 5× less memory (1.7 GB vs 9.2 GB), with lower reconstruction error.

Hybrid 3D VAE Mirrors Figure 2 of the paper. The ground-truth 256 cubed TUDF goes through a sparse encoder (256, 128, 64, 32 cubed) and is densified into a 16 cubed latent z, regularised by a KL term. The decoder samples z, upsamples densely to 32 and 64 cubed, predicts occupancy, sparsifies, and upsamples sparsely to 128 and 256 cubed to give the reconstruction. Losses: BCE plus Dice on occupancy against the downsampled ground truth, and L1 on the reconstruction. Dashed blocks are sparse, solid blocks dense; visible surfaces keep the photo colour and hidden ones are blue. ground truth Vgt 256³ 128³ 64³ 32³ Encoder densify latent z ℒKL 32³ 64³ 128³ 256³ occupancyhead Decoder sample sparsify reconstruction V̂ downsample BCE + Dice ℒ1 sparse block dense block visible hidden
The same encoder, latent and decoder as above, drawn as in Figure 2 of the paper (NRGB-D kitchen; the reconstruction is the real VAE output, 93.5% surface IoU).

04Generation

Generate the latent from a single image

A diffusion transformer, trained with flow matching, generates the VAE latent from noise in 50 steps. The VAE decoder then turns that latent into the TUDF grid. The transformer is conditioned on the input image in two ways:

  • 1Global geometric prior. Frozen MoGe-2 features enter every transformer block through cross-attention, giving the overall layout and scale.
  • 2Visible-latent inpainting. MoGe-2's visible point map is converted to a TUDF, encoded by the frozen VAE encoder into $z_{\text{vis}}$, and added to the noisy latent through a zero-initialized layer: $\tilde z_t = z_t + \mathrm{MLP}_{\text{zero}}(z_{\text{vis}})$. This anchors the result to what the camera actually saw.
VolFill generation pipeline Mirrors Figure 3 of the paper. The input image goes to a frozen MoGe-2. Path 1: its image tokens enter every block of the denoising transformer through cross-attention. Path 2: its visible geometry is encoded by the frozen VAE encoder into a visible latent, passed through a zero-initialised MLP and added to the noisy latent. The trained transformer runs for 50 steps; the frozen VAE decoder turns the final latent into the amodal output, where visible surfaces keep the photo colour and hidden ones are blue. Snowflakes mark frozen modules, flames trained ones. input image MoGe-2 image tokens cross-attention 1 2 visible geometry Encoder visible latent zvis MLPzero noisy latent zt z̃t DiTblock DiTblock ··· DiTblock denoising transformer Φ timestep t × 50 steps Decoder amodal 3D output frozen trained visible hidden
Trained on 3D-FRONT (96k image–TUDF pairs) and ScanNet++ (46k samples).

Try it

Step through a real generation

At each of the 50 steps, we decode the model's current estimate of the final latent and show it as a point cloud. Drag $t$ to watch the scene take shape.

Denoising trajectory loads here.

05Qualitative

Compare with other methods

Pick a scene and a baseline. The baseline is shown next to VolFill. Cameras are synced: orbit either one and look behind the furniture. Colours follow the paper, judged from each method's own geometry: visible surfaces keep the photo's colour, hidden ones are blue.

Input RGB image
Input image
MoGe2
VolFill (ours)

Keys: ← → scene · 1–4 baseline · R reset · F fullscreen

From the paper

VolFill synthesizes sharp, high-fidelity amodal geometry. LaRI produces layered artifacts (red circles) and holes, and NOVA3R yields noisy, unstructured point scatters (green circles).

06Quantitative

Pixel-aligned models win on visible surfaces, which they are built for. On the occluded and complete subsets, VolFill leads on both benchmarks.

Occluded coverage (APD) and complete F-score at the tight threshold, higher is better. VolFill in yellow.
Table 1 · 3D reconstruction on SCRREAM (×10²)
Method Visible Occluded Complete
CD ↓APD0.1 ↑APD0.02 ↑ CD ↓APD0.1 ↑APD0.02 ↑ CD ↓FS0.1 ↑FS0.02 ↑
Object-level generative†
TripoSG†13.7544.7410.0810.7355.3413.3711.1053.5613.30
TRELLIS†11.6354.0515.009.6560.7017.5210.7256.6915.59
Pixel-aligned
VGGT3.0496.9845.6217.4044.6010.046.7780.4536.77
DepthAnything32.3798.9854.3416.1648.3110.766.1682.8643.10
DepthPro3.8595.1033.3015.3148.669.716.9080.3628.44
MoGe22.2899.3857.4215.2950.0910.305.7483.9645.84
Scene-level amodal
LaRI3.4395.4841.615.2385.2529.315.1985.4330.85
NOVA3R3.2096.7743.753.5694.8641.213.4395.2645.12
VolFill (ours)2.8496.1056.733.4692.7055.533.0395.0354.83

† Trained on object-centric data; not designed for full scenes.

Metric-scale geometry

Without Sim(3) alignment. FPD is the Fréchet distance between Uni3D features of predicted and real point clouds, so it measures how realistic the generated geometry is as a distribution.

Table 3 · Metric geometry evaluation
MethodCD ↓FS0.02 ↑FPD ↓
LaRI11.4311.47—
NOVA3R6.5622.2316.50
VolFill (ours)3.8844.904.86

07Ablations

Both conditions matter

The visible latent alone (①) cannot recover unobserved structure. Image-only conditioning with DINOv2 (②) or MoGe-2 (③) hallucinates rough layouts but loses detail. The dual setup (④), MoGe-2 cross-attention plus additive visible latent, gives the sharpest, most complete geometry.

Table 5 · Conditioning
Config.FGFMzvisCD ↓FS0.02 ↑
Single✗Add6.8031.72
MoGe2✗3.8146.80
DualMoGe2Concat3.6648.94
MoGe2Add3.0354.83
Table 6 · Foundation model
ConditioningCD ↓FS0.02 ↑
DINOv25.4635.98
VGGT4.3242.30
MoGe23.8146.80

Swapping MoGe-2 for the semantic-only DINOv2 collapses quality, because semantic tokens lack the 3D anchors the DiT needs. VGGT recovers part of the gap. A geometry-trained backbone is a stronger prior for amodal completion than general semantic features.

08Limitations

  • Residual holes in unobserved regions. VolFill produces sharper geometry with fewer artifacts than NOVA3R and LaRI, but small holes can persist behind heavily occluded surfaces. Synthesizing geometry with no visual evidence is fundamentally underdetermined; more model capacity and data diversity are the natural levers.
  • Iterative inference. 50 Euler steps with classifier-free guidance take about 1.4 s on an RTX 4090, slower than single-pass regression. Step distillation and consistency-style training are promising ways to speed this up.

09Cite

BibTeX

@inproceedings{ngo2026volfill,
  title     = {VolFill: Single-View Amodal 3D Scene Reconstruction
               with Volumetric Flow Matching},
  author    = {Ngo, Tuan Duc and Gan, Chuang and Kalogerakis, Evangelos},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026}
}

References

  1. Wang et al. MoGe-2: Accurate Monocular Geometry with Metric Scale and Sharp Details. NeurIPS 2025.
  2. Lin et al. Depth Anything 3: Recovering the Visual Space from Any Views. ICLR 2026.
  3. Bochkovskii et al. Depth Pro: Sharp Monocular Metric Depth in Less Than a Second. ICLR 2025.
  4. Wang et al. VGGT: Visual Geometry Grounded Transformer. CVPR 2025.
  5. Li et al. LaRI: Layered Ray Intersections for Single-view 3D Geometric Reasoning. arXiv 2025.
  6. Chen et al. NOVA3R: Non-pixel-aligned Visual Transformer for Amodal 3D Reconstruction. ICLR 2026.
  7. Li et al. TripoSG: High-Fidelity 3D Shape Synthesis using Large-Scale Rectified Flow Models. arXiv 2025.
  8. Xiang et al. Structured 3D Latents for Scalable and Versatile 3D Generation (TRELLIS). CVPR 2025.
  9. Jung et al. SCRREAM: SCan, Register, REnder And Map. A Framework for Annotating Accurate and Dense 3D Indoor Scenes with a Benchmark. NeurIPS 2024.
  10. Azinović et al. Neural RGB-D Surface Reconstruction (NRGB-D). CVPR 2022.
  11. Zhou et al. Uni3D: Exploring Unified 3D Representation at Scale. ICLR 2024.