PlaneFormers: From Sparse View Planes to 3D Reconstruction

Abstract

Indoor scenes such as rooms, kitchens and other areas of houses are pre-dominantly composed of planar surfaces. These surfaces may thus be used to generate high-quality scene reconstructions. Previous approaches in this domain are generally optimization-based and use an expensive bundle adjustment step. We propose the PlaneFormer, a transformer-based approach, that takes in 3D-aware plane tokens as inputs and can reconstruct scenes using planar surfaces. Our experiments show that our method outperforms baselines on most metrics, can be extended reasonably to multiple views and requires several 3D-specific choices to perform effectively on the task.