[Tech] Structure-from-Motion: COLMAP

by Geunho Jung (AI researcher / R&D)

 

Our RebuilderAI R&D team is introducing COLMAP this week.

Structure-from-Motion: COLMAP | RebuilderAI

Structure-from-Motion: COLMAP | RebuilderAI

 

More available information is here: Home | RebuilderAI

Home | RebuilderAI

Before introducing the technology, let me tell the overall 3D Reconstruction Pipeline.

[attachment1] 3D conversion pipeline

If we put a 360 video or image, we must first calculate the camera information before entering the AI model. After calculating the camera information, the image and the camera information corresponding to the image are input to the AI model and converted into 3D. To reconstruct to 3D, it needs camera information for each image, and the required module is COLMAP. 

 

COLMAP is often used to reconstruct 3D from images taken from multiple points in time. We can calculate camera information by inserting images into COLMAP, and furthermore, can obtain 3D model.

 

Today, we're going to talk about one of the 3D conversion pipelines, the software used for camera information calculation and 3D reconstruction, called COLMAP.

 

1. Structure-from-Motion

2. Wrap up

---

1. Structure-from-Motion

[attachment2] Structure-from-Motion pipeline

COLMAP is a program implementation of a technology called Structure-from-Motion.

Structure-from-Motion (SfM) is a system that reconstructs a three-dimensional structure from images taken at multiple points in time for a single scene. SfM consists of three main steps. 

 

(1) Feature detection

(2) Feature matching

(3) Structure / Motion reconstruction

 

---

(1) Feature detection

[attachment3] Feature example - red part

"Feature" is the characteristic part of an image that can be described. As shown in [attachment3], the parts which have a large change or characteristic part compared to the surroundings, such as the eyes and hair on the human face can be "features". On the contrary, the parts that do not change or stand out such as the shoulders and cheeks of a person are the parts with fewer "features". Because the SfM system starts with the detected feature, it doesn't function properly when images with fewer features are entered. For this reason, we can fail the conversion or cannot get proper results if we put an image that does not show prominent features. ([attachment4])

 

[attachment4] Example of images with fewer features

 

 

(2) Feature Matching

 

After detecting features in each image, you will go through the process of matching the features in each image. Feature matching is to determine the part where the features of one image and another are matched. ([attachment5])

[attachment5] Example of Feature matching

As you can see in [attachment5], you can see that the same features are matched and connected. Also, the matched points can be calculated as three-dimensional points by triangulation and can calculate the Camera pose. A Camera pose is a Matrix that indicates where the camera is located. This matrix represents how rotated the camera is (R) and where it is (T or t), and is essential information in the field of 3D reconstruction. (More information about Camer will be provided in the next tech blog.)

[attachment6] Example of Camera pose

However, these matched points may have an Outlier. These can cause errors in the result, so we need to eliminate them. Various algorithms for optimization eliminate outliers can leave only exactly matched points. 

 

(3) Structure / Motion reconstruction

[attachment7] Point cloud and Camera(redpoint) from COLMAP

[attachment8] Point cloud and Camera(redpoint) from COLMAP (2)

[attachment9] Point cloud and Camera(redpoint) from COLMAP (3)

 

---

2. Wrap up

We introduced each step in COLMAP which is widely used for Camera pose calculation and 3D reconstruction, briefly introduced what features in the image are and what processes are used to obtain information necessary for 3D reconstruction. Although there is various SfM-based software such as VisualSFM and OpenMVG, we have learned about COLMAP, currently widely used in 3D reconstruction research. Next technique contents will be algorithms and camera information used at each stage. Thank you. 

 

---

Reference

[attachment2]: https://docs.opencv.org/3.4/dc/dc3/tutorial_py_matcher.html
[attachment3]: https://it.mathworks.com/matlabcentral/fileexchange/50319-sift-feature-extreaction
[attachment5]: https://docs.opencv.org/3.4/dc/dc3/tutorial_py_matcher.html
[attachment6]: https://www.researchgate.net/figure/Homography-and-Planar-Motion-The-two-observations-of-a-3D-point-lying-on-a-plane-are_fig1_221362899
[attachment7]: https://colmap.github.io/index.html
[attachment8]: https://www.researchgate.net/figure/The-results-of-COLMAP-reconstruction-on-rectified-images-similar-to-those-shown-in-Fig_fig4_345555520
[attachment9]: https://www.wenyanet.com/opensource/ko/61a778dcabe1f82dec18dfe8.html