Nvidia's Neuralangelo AI model can create 3D models from videos

Nvidia recently revealed its new AI model, Neuralangelo, for reconstructing 3D models from video footage. 

Nvidia's Neuralangelo AI model can create 3D models from videos

Nvidia's Neuralangelo AI model can create 3D models from videos

Neuralangelo is designed to overcome the difficulties encountered by traditional photogrammetry tools by using a neural network to generate and progressively optimize a volumetric representation of a scene from a set of source images. 

Nvidia claims that Neuralangelo can reconstruct surface detail more accurately than existing approaches, and will be presented at the CVPR 2023 conference this month.

Nvidia Neuralangelo Release Date

Despite its promise, Nvidia has yet to release source code for Neuralangelo, so it is currently an indication of how software may develop in the future rather than a practical production tool. 

In this article, we'll take a look at what Neuralangelo is and how it differs from existing neural surface reconstruction methods.

What is Neuralangelo?

Neuralangelo is the result of Nvidia’s latest work in neural surface reconstruction, an AI-based alternative to the multi-view stereo (MVS) approach used in traditional photogrammetry tools. 

The new methods train a neural network to generate and progressively optimize a volumetric representation of a scene from a set of source images.

The neural surface reconstruction approach avoids some of the drawbacks of MVS – like its difficulty in resolving regions of an object with solid color variation or large areas of homogenous color. However, current methods still struggle to recover detail from real-world scenes.

How Does Neuralangelo Differ from Existing Neural Surface Reconstruction Methods?

Neuralangelo uses Instant Neural Graphic primitives (Instant NGP), the same neural representation of the underlying 3D scene used in its downloadable Instant NeRF toolset. 

The representation is refined using a new coarse-to-fine optimization strategy that Nvidia’s blog post compares to a sculptor chipping away at a stone block to create a statue.

The metaphor is picked up in the name of the AI model, and in one of the objects reconstructed for its demo video: Michaelangelo’s David. Nvidia suggests that this approach results in a “complexity and fidelity [that] far surpass prior methods”.

How is Neuralangelo Better than Existing Techniques?

Nvidia says that Neuralangelo reconstructs surface detail more accurately than existing approaches: a claim supported by the mathematical analysis in its research paper Neuralangelo: High-Fidelity Neural Surface Reconstruction.

You can gauge the results visually from the video footage and the close-up images in the paper. As well as the statue of David, Nvidia’s test objects include a flatbed truck and its own Santa Clara campus reconstructed by Neuralangelo from drone footage.

When Will I be Able to Use Neuralangelo?

Unfortunately, Nvidia has yet to release source code for Neuralangelo, so it’s currently an indication of how software may develop in the future rather than a practical production tool. When asked whether it would release the code in the future, Nvidia said that it “[did] not comment on unannounced research or code”.

Nvidia Neuralangelo is currently an exciting but elusive tool, offering a potential glimpse of the future of 3D reconstruction from video footage. Until Nvidia releases its code, however, it remains just a potential. {Stop article}