NVIDIA new AI "magic": Turn 2D photos into 3D models

NVIDIA new AI “magic”: Turn 2D photos into 3D models

NVIDIA new AI “magic”: Turn 2D photos into 3D models.

At this week’s Conference on Computer Vision and Pattern Recognition (CVPR), Nvidia unveiled an AI “magic” called 3D MoMa: turning static 2D photos into easily editable 3D models.

According to NVIDIA, 3D MoMa allows architects, designers, artists and game developers to quickly import 3D models converted from images into a graphics engine, and to modify scale, change materials or try different lighting effects during use.

1. Extract 3D models from 2D images

In reality, for an artist or engineer to get the most out of a 3D model, it’s best to put it in a widely used tool such as a game engine, 3D modeler, or movie renderer, but only if the 3D model is A triangular mesh form with textured materials (primarily used to define the basic framework for 3D graphics and modeling shapes).

Typically, many game studios and creators habitually use complex photogrammetry techniques to create such 3D models, which are laborious and time-consuming to do manually – but the advent of 3D MoMa may change this situation.

It is understood that behind 3D MoMa is a reverse rendering technology called “the holy grail of unified computer vision and computer graphics” by David Luebke, vice president of graphics research at NVIDIA.

Inverse rendering, the technique of reconstructing a series of static 2D photos into a 3D model of an object or scene, 3D MoMa uses modern AI machines and NVIDIA GPUs by defining each inverse rendering problem as a GPU-accelerated differentiable component.

Raw computing power to rapidly generate 3D objects, allowing creators to import, edit and extend them without limitations within existing tools.

According to NVIDIA’s official blog, 3D MoMa can generate a triangular mesh model within 1 hour on a single Tensor Core GPU, and the output will be directly compatible with the 3D graphics engine and modeling tools the creator is using.

Not only that, because 3D MoMa generates a triangular mesh model, and the mesh is like a paper model of a 3D shape built from triangles, so creators can rebuild it: changing its mesh model, materials, and lighting effects.

2. Virtual 3D musical instrument show jazz performance

To more intuitively demonstrate the capabilities of 3D MoMa, the NVIDIA research team first collected nearly 100 images of five jazz band instruments (trumpet, trombone, saxophone, drum kit, and clarinet) from different angles.

The research team then used 3D MoMa to convert these 2D images into triangular mesh models of each instrument, which were taken from the original scene and imported into the Omniverse 3D simulation platform for editing.

As mentioned above, creators can reconstruct the output of 3D MoMa, so the NVIDIA research team also tried it.

The material can be changed, and the original plastic material of the trumpet can be converted into gold, marble, wood or cork;
The newly edited model can be placed in a virtual scene, and these instruments are placed in a classic Cornell Box for graphics rendering quality testing, and it is found that the virtual instruments reflect light in the same way as in the real world;
These new models, generated by reverse rendering, can be used as building blocks for complex animation scenes, as shown in the video below.

While the current 3D MoMa is still in development, Nvidia is looking forward to it: I believe it will allow game developers and other designers to quickly modify 3D objects and then add them to any virtual scene.

Reference link:

https://blogs.nvidia.com/blog/2022/06/21/inverse-rendering-3d-research-cvpr/

https://thenextweb.com/news/nvidias-new-ai-magic-turns-2d-photos-into-3d-graphics

DIY a Router

DIY a PBX