An Empty Room is All We Want:
Automatic Defurnishing of Indoor Panoramas

We propose a pipeline that leverages Stable Diffusion to improve inpainting results in the context of defurnishing---the removal of furniture items from indoor panorama images. Specifically, we illustrate how increased context, domain-specific model fine-tuning, and improved image blending can produce high-fidelity inpaints that are geometrically plausible without needing to rely on room layout estimation. We demonstrate qualitative and quantitative improvements over other furniture removal techniques.

Our pipeline consists of the following components:

• Pre-processing: Furniture mask estimation via semantic segmentation & image rolling and padding to ensure optimal context, and downsampling to fit into a Stable Diffusion pipeline.
• Inpainting: Our custom inpainting, fine-tuned on equirectangular panoramas and robust to inexact masks and remnant shadows, thus reducing the tendency of Stable Diffusion inpainting to hallucinate objects.
• Post-processing: Super-resolution and blending of the original and inpainted images, so that high-frequency details are preserved.

The paper has been accepted to the Workshop on Generative Models for Computer Vision at CVPR 2024. A short version has been accepted to the Workshop on Computer Vision in the Built Environment at CVPR 2024.

@inproceedings{matterport2024defurnishing,
    author    = {Slavcheva, Mira and Gausebeck, Dave and Chen, Kevin and Buchhofer, David and Sabik, Azwad and Ma, Chen and Dhillon, Sachal and Brandt, Olaf and Dolhasz, Alan},
    title     = {An Empty Room is All We Want: Automatic Defurnishing of Indoor Panoramas},
    booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
    year      = {2024},
}