HyperNeRF: Three-Dimensional Models from Moving Camera Images

Here is the paper, “A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields”. The abstract is as follows.

Neural Radiance Fields (NeRF) are able to reconstruct scenes with unprecedented fidelity, and various recent works have extended NeRF to handle dynamic scenes. A common approach to reconstruct such non-rigid scenes is through the use of a learned deformation field mapping from coordinates in each input image into a canonical template coordinate space. However, these deformation-based approaches struggle to model changes in topology, as topological changes require a discontinuity in the deformation field, but these deformation fields are necessarily continuous.

We address this limitation by lifting NeRFs into a higher dimensional space, and by representing the 5D radiance field corresponding to each individual input image as a slice through this “hyper-space”. Our method is inspired by level set methods, which model the evolution of surfaces as slices through a higher dimensional surface. We evaluate our method on two tasks: (i) interpolating smoothly between “moments”, i.e., configurations of the scene, seen in the input images while maintaining visual plausibility, and (ii) novel-view synthesis at fixed moments. We show that our method, which we dub HyperNeRF, outperforms existing methods on both tasks. Compared to Nerfies, “HyperNeRF” reduces average error rates by 4.1% for interpolation and 8.6% for novel-view synthesis, as measured by LPIPS.

Python source code is available on GitHub.

Here is the video that accompanied the paper.


Back in the stone age of electron microscopy - around 1972 - I was examining thin sections of the superior oblique muscle of the newborn rat in the electron microscope. The result was two-dimensional images. We were trying to figure out the sub-cellular three-dimensional architecture of the muscle cells. Now, in normal microscopy, one could pretty easily make serial sections through a tissue or organ and reconstruct three-dimensions at that much larger level. Doing so with serial sections through such ultra thin sections through a single or a few cells was quite difficult.

What I recall is that there were, even then, mathematical models which allowed us to calculate the volume of sub-cellular, two-dimensional images of organelles by applying formulas. At the time, this was the best we could do. Maybe this is a descendant of that early work. I don’t remember the details.