BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling

Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, Anton van den Hengel
Amazon
Paper Code (coming soon) Data (coming soon)

BLiRF can model challenging scenes with light/texture changes, long-range motion, and object warpings.

Abstract

Reasoning the 3D structure of a non-rigid dynamic scene from a single moving camera is an under-constrained problem. Inspired by the remarkable progress of neural radiance fields (NeRFs) in photo-realistic novel view synthesis of static scenes, extensions have been proposed for dynamic settings. These methods heavily rely on neural priors in order to regularize the problem. In this work, we take a step back and reinvestigate how current implementations may entail deleterious effects, including limited expressiveness, entanglement of light and density fields, and sub-optimal motion localization. As a remedy, we advocate for a bridge between classic non-rigid-structure-from-motion (\nrsfm) and NeRF, enabling the well-studied priors of the former to constrain the latter. To this end, we propose a framework that factorizes time and space by formulating a scene as a composition of bandlimited, high-dimensional signals. We demonstrate compelling results across complex dynamic scenes that involve changes in lighting, texture and long-range dynamics.

BLiRF

BLiRF offers a theoretical framework that enables efficient integration of implicit neural priors and well-defined explicit priors (inspired by NRSFM). Thus, BLiRF can be considered as a bridge between NeRF and NRSfM. In particualr, BLiRF model the light and density fields of a 3D scene as bandlimited, high-dimensional signals. This standpoint enables complete factorization of spatio- temporal dynamics, allowing us to inject explicit priors on the time and space dynamics independently.


Evaluation on topologically varying scenes

Our formulation implicitly enables effective modeling of topologically varying scenes. We obtain comparable results against the state-of-the-art and use significantly less compute resources, e.g. 64 hours on 4 TPU v4s (HyperNeRF) vs 3 hours on a single V100 (ours).

-->

Novel view synthesis results on the Nvidia Dynamic Scene dataset.

Our method achieves state-of-the-art results.

Geometry

BibTeX

@article{ramasinghe2023bali,
  author    = {Ramasinghe, Sameera and Shevchenko, Violetta and Avraham, Gil and Hengel, Anton Van Den},
  title     = {BLiRF: Bandlimited Radiance Fields for Dynamic Scene Modeling},
  journal   = {arXiv preprint arXiv:2302.13543},
  year      = {2023},
}