Instant NeRF AI has learned how to turn 2D photos into 3D scenes

Instant NeRF AI has learned how to turn 2D photos into 3D scenes.

NVIDIA has developed Instant NeRF, a rendering neural model that learns a high-resolution 3D scene in seconds and can render images of that scene in a few milliseconds.

The model uses an AI-assisted process known as inverse rendering to determine the behavior of light in the real world, allowing researchers to reconstruct a 3D scene from multiple 2D images taken from different angles.

NVIDIA has shown that the model solves this problem almost instantly, making it one of the first to combine ultra-fast neural network training and fast rendering.

NVIDIA has taken this approach to a technology called Neural Radiation Fields, or NeRF. The implementation of the technology in some cases reaches more than 1000-fold acceleration. It only takes a few seconds for a model to learn from a few dozen 2D photographs and the camera angles they were taken from, and then it can render the resulting 3D scene in tens of milliseconds.

Collecting data for NeRF transmission is a bit like a red carpet photographer trying to capture a celebrity outfit from all angles: the neural network needs several dozen images taken from different points of the scene, as well as the camera position of each of them. For a scene that has people or other moving elements, the faster these shots are taken, the better. If there is too much movement during 2D image capture, the AI-generated 3D scene will be blurry.

In a scene, NeRF essentially fills in the gaps by training a small neural network to reconstruct the scene by predicting the color of light emitted in any direction from anywhere in 3D space. This method can even work with occlusions - where objects visible in some images are blocked by obstacles such as pillars in other images.

While estimating the depth and appearance of an object based on a partial representation is a natural skill for humans, it is a difficult task for AI. Creating a 3D scene using traditional methods takes several hours or more, depending on the complexity and resolution of the rendering. The introduction of AI speeds up work. Early NeRF models rendered crisp, artifact-free scenes in minutes, but took hours to train.

However, Instant NeRF reduces rendering time by several orders of magnitude. It is based on a technique developed by NVIDIA called Multi-Resolution Hash Grid Encoding, which is optimized to run efficiently on NVIDIA GPUs. Using a new input encoding method, researchers can achieve high-quality results using a small and fast neural network

The model was developed using the NVIDIA CUDA toolkit and the Tiny CUDA neural network library. Because it is a lightweight neural network, it can be trained and run on a single NVIDIA GPU and runs fastest on cards with NVIDIA Tensor Cores.