[Tech] Object Recognition | SOD & TOD

by Sanghyeon An (AI research engineer / R&D)

 

Hello. We are the RebuilderAI, which provides a technology platform based on 3D modeling automatic AI solution. Today we are sharing 리빌더AI 물체 인식 기술 | RebuilderAI

리빌더AI 물체 인식 기술 | RebuilderAI

More available information is here: Home | RebuilderAI

Home | RebuilderAI

 

 

국문(Korean) ver: https://blog.naver.com/rebuilderai/222811359320

RebuilderAI | 메타버스의 시작, 물체 인식 기술

---

Table of Contents

1. Object Recognition Problem

2. SOD(Salient Object Detection)

3. Transparent Area Recognition Problem

4. TOD(Transparent Object Detection)

 

---

1. Objection Recognition Problem

We'd like to restore objects that are common around us just by taking videos with smartphones regardless of the type.

But in this case, both objects and background are taken in the video. Then how can we reconstruct only the objects that we want to make in 3D?

Think simply, remove all the rest except for the object.

We’d like to solve these problems using the technique called “SOD(Salient Object Detection)”.

 

 

---

2. SOD(Salient Object Detection)?

[attachment1]

‘Salient’ means ‘most important’, so ‘Salient Object Detection’ is a matter of ‘finding the most important object’.

Then let me show you an example. In this situation, it is detecting the most important object in the video. In other words, it’s about finding the main object in that image. Then treat the rest as the background and remove it. Look at the picture below.

 

[attachment2]

This is the detection of plant that exists in our space using Salient Object.

These tasks are somewhat different from general object detection or segmentation. General object detection or segmentation has a class, and other objects that do not belong to that class cannot be detected. So, it is hard to scan any object and make it 3D.

That’s why we selected the SOD model, not detection or segmentation.

 

*General Object Detection: Classify the image and identify the object.

 

 

 

---

3. Transparent Area Recognition Problem

But that’s not all.

Reconstructing transparent areas is one of the traditional problems in 3D reconstruction.

Many 3D reconstruction models use ‘depth’, but in the case of transparent areas, reconstruction is not done properly because depth isn’t calculated well. The reason that depth is not calculated properly is that the background is transparent. In general, depth is calculated using the difference between the current location and image values viewed from a different location. But it cannot be calculated when the back is visible.

You can see the result in the figure below.

[attachment3]

If you look at the result, other opaque objects are reconstructed well while transparent objects are less reconstructed than those that are.

It is a problem that is being studied actively to reconstruct transparent areas. Many studies have been conducted to solve these problems, and many of them have attempted to use properties such as ‘reflection’ and ‘refraction’.

We thought we need to treat transparent areas as exceptions and apply different algorithms to detect transparent areas. These issues are classified as TOD(Transparent Object Detection).

 

 

---

4. TOD(Transparent Object Detection)

The above picture is the bottle we are using, and it is the result of the original object and the SOD/TOD object. To show a transparent area has been detected well, we set the mask value low to make it look dark.

 

If there is a transparent part like the bottle above, you should follow this way.

 

(1) Detect transparent area separately
(2) Separate opaque areas and masks
(3) Send it to the 3D reconstruction parts

 

These processes are included in the preprocessing pipeline for 3D reconstruction to ensure that objects can be reconstructed as neatly as possible.

 

 

---

Reference

source of [attachment1]: https://techxplore.com/news/2020-12-salient-vision-smarter.html

Salient object detection makes computer vision smarter

source of [attachment3]: https://ai.googleblog.com/2020/02/learning-to-see-transparent-objects.html 

Learning to See Transparent Objects