[Tech] Surface Reconstruction with Implicit Representation

by Geunho Jung (AI researcher / R&D)

 

Hello. This is RebuilderAI R&D Team.

This week, we are introducing Surface reconstruction technology based on Deep learning. 

https://rebuilderai.github.io/reconstruction/2022/07/26/IDR.html

Surface Reconstruction with Implicit Representation | RebuilderAI

More available information is here: Home | RebuilderAI

Home | RebuilderAI

Like the NeRF we posted last time, the images and camera parameters taken at various points in time are used as input, but it differs in the purpose that Surface reconstruction is to reconstruct the 3D surface well.

The technology I'm going to introduce today is the method proposed in the IDR: Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance paper introduced at the 2020 Neurlps.

 

---

Table of Contents

1. 3D geometry representation

2. Implicit Neural Representation

3. Differentiable Intersectino of Viewing Direction and Geometry

4. IDR Result

5. Wrap up

 

---

1. 3D geometry representation

 

Recently, to solve the problem of reconstructing 3D shapes from 2D images in computer vision/graphics, the Differentiable rendering method is used a lot, and the optimization is mainly used to express 3D geometry well. In this case, there are ways to express 3D geometry such as Point cloud, Triangle mesh, and Implicit representation. There are Volumetric grid (NeRF) and Zero level set (SDF) ways to represent Implicitly.

[attachment1] Point cloud and Triangle mesh

 

In IDR, they use Implicit zero level set (SDF) method to express 3D geometry. Through this Implicit method, 3D surface with arbitrary shape and Topology can be more flexible and well expressed. Now, I’ll explain in more detail how 3D geometry is Implicitly expressed based on deep learning and how Appearance is handled.

 

---

2. Implicit Neural Representation

First, let me tell you more about Implicit representation based on deep learning. Typical Neural network Parameterized based on discretized information (e.g. predict Depth given pixels in an image). Conversely, Implicit (neural) representation can Parameterize continuous information. For example, a function that maps an image to the corresponding RGB given the x, y coordinates on the image is an example of an implicit representation. In other words, learning a Neural network for Implicit representation means learning the parameters of the Neural network so that it is a function of representing an image when given a coordinate (for example, an image).

 

IDR used this Implicit representation concept to design a Neural network architecture that allows to output corresponding Shape and Appearance when given a 3D coordinate. The structure of IDR consists largely of Geometry (Implicit neural representation) network, Sampling network, Neural renderer (Appearance network). Let me introduce each one now.

 

- Geometry

 

First, it is a network that expresses Geometry.

In this network, they model SDF to Implicit Zero level set (Implicit representation). Here, SDF is a function of the distance from the Surface given a coordinate, which is expressed by a sign whether the coordinate is inside or outside the Surface, and if Surface, it is a function of 0.

[attachment2] Surface SDF; red is eternal(+), blue is internal(-)

In this way, when the network is given coordinates, it learns Implicit representation so that it can express the exact Surface (i.e. SDF = 0).

 

---

3. Differentiable Intersection of Viewing Direction and Geometry

 

IDR is a Differentiable rendering way based on deep learning. Therefore, when applying the Surface point (Intersection point) in the Geometry network, they do a Sampling after calculating whether the expression for obtaining the Intersection point and the initial parameter match exactly with the Intersection point through the Geometry network. This process was implemented as a Sampling network. In this process, you need a Normal vector, which is calculated and used by differentiating the differentiable depth (SDF).

 

- Appearance

 

Finally, let me explain how IDR models Surface’s Radiance.

 

IDR wanted to use the Bidirectional Reflectance Distribution Function (BRDF) to express how light is reflected and emitted when it enters the Surface. BRDF is a function of predicting the reflectance when you know the angle at which light comes in, the angle at which it is reflected, and the normal at that Surface. At this point, assume the angle at which the light is reflected reflects the light and enters the camera. You can calculate the amount of light reaching the camera through the Rendering equation using the light emitted from the Surface and the reflectance predicted by the BRDF and the incoming light. IDR models this Rendering equation through Neural network to predict the final Color view.

 

- Rendering Equation

 

Rendering equation (Equation 1) is a widely used formula in Renderer in computer graphics. Let me briefly explain this equation.

[equation1] Rendering equation

It looks a little complicated, but in a simple way, it’s as follows.

The result of the integration here is reflected light. (The factor that controls the amount of light in this equation is used to mathematically define when the amount of light coming in is smaller or more depending on the slope. In other words, if the angle increases, the amount of light coming in decreases, and if the angle decreases, the amount of light gets larger.)

In addition, reflexibility can be obtained through the BRDF, which is a function that calculates the reflectance, the probability of reflection at a given angle of incidence and reflection angle.

 

---

4. IDR Result

 

[attachment3] Reconstruct results through IDR

[attachment3] shows the results of the reconstruction using the IDR method. It’s the result of using public dataset DTU images that are commonly used in 3D reconstruction, and it shows that they are better reconstructed than a traditional method based on Colmap, rather than DVR or deep learning-based methods.

 

---

5. Wrap up

 

IDR is a deep learning-based method presented for 3D reconstruction. I could see that it was reconstructed well with only images and camera information. Also, it was a good technology in that it considered reflectance by modeling the Rendering equation used in computer graphics. It’s a little bit of an older method published in the 2020 NeurIPS, but I think we can reach better reconstruction performance by looking at the ideas and design approaches we’ve considered here.

 

I will end the introduction of IDR technology here. Thank you.

 

---

Reference

[attachment1]: https://towardsdatascience.com/3d-object-classification-in-6-steps-using-kaolin-and-colab-9ecb079143a8

[attachment2]: Implicit Geometric Regularization for Learning Shapes

[attachment3]: https://lioryariv.github.io/idr/