How to improve image segmentation accuracy with AI

Hello, in this article, I'll share a real-world example of how DataHunt utilizes ImageSegmentation technology in our data processing work, specifically how we use masks and polygons, and finally, what the results were.

What is ImageSegmentation?

Image Segmentation isa technology that classifies objects or backgrounds in an image on a pixel-by-pixel basis. It is also called dense label because it is labeled at the pixel level, which is the smallest unit of an image. And this technology can be divided into Semantic segmentation, Instance segmentation, and Panoptic segmentation according to the way of separating areas.

‍

‍

‍

Here's a short description of each

‍

• Semantic Segmentation: Categorizing all pixels in an image into predefined classes. The concept of classification does not distinguish between objects of the same class and recognizes them as one if they overlap.
• Instance Segmentation: Classifies only objects in an image, recognizing objects of the same type separately from each other.
• Panoptic Segmentation: A combination of the above two, also called thing (countable objects) and stuff (uncountable objects).

And each labeled thing is called a mask.

So how can a human create the above masks? Without the help of AI, there are two main ways to create them by hand.

‍

• Create them by coloring them in, like the brush feature in Paint
• Create a polygon that surrounds the object.

However, using brushes can be cumbersome because you have to color each one individually, and creating a polygon can be cumbersome because you have to take a lot of points.

‍

‍

Which feature is better to utilize? After many experiments with both brushes and polygons, we decided that polygons were better in terms of accuracy and time, but they were relatively good, and with so many points for a single car, we needed AI to help us with tens of thousands or hundreds of thousands of images.

How to apply AI to image segmentation

Which image segmentation job can be done by AI?

There are several ways to use AI to obtain masks or polygons. Here are some of them

Obtaining a mask

• Get a mask from a segmentation model
• Click on an object in an image to get a mask
• Get a mask by creating a bounding box that contains the image
• Get a mask by clicking four points on the top, bottom, left, and right of an object.

How to get polygons

• Obtain polygons through a model that infers polygons
• Apply an algorithm to convert the obtained mask to polygons

and so on. Datahunt is equipped with several of the above methods, but the method we will introduce this time is an algorithm that converts the obtained mask into polygons. The reason for this is that, as explained earlier, we obtained statistics that it was easier for workers to work with polygons than brushes, so we decided that it would be most effective to deliver better polygons through AI.

AI application and challenges

‍

But even AI can't be all things to all people. While we were able to get a polygon that surrounded the car, as shown in the image above, the problem was not its accuracy.

‍

• We need to take theAI's results and modify them (create/move/delete points).
• Too many points

was the problem.

While it's nice to have a lot of dots, it's a laborious process, so we needed a way to effectively reduce the number of dots.

How to improve the application of AI to image segmentation

Tilt-based filtering

The first method is to reduce the number of points by utilizing the change in tilt.

‍

In the image above, you can see that we initially have six dots in what appears to be a nearly straight line. But we don't really need that many, so we're going to reference the slope one by one from an arbitrary starting position and remove the points where the change is small.

Proportion-based filtering

The next method is ratio-based filtering, which directly reduces the number of points by taking into account the size and curvature of the object.

‍

1. divide the object area into small grids
2. calculate the curvature of a point for every grid it exists in
3. determine a sampling rate proportional to the object size and curvature
4. reduce the number of points in each grid based on the determined sampling rate
   ‍

What do you get after the above two filtering methods? Obviously, if the polygons still enclose the object well despite reducing the number of points, it's a success. Below is an illustration of the process of getting the mask, converting it to polygons, and reducing the number of points from the left.

‍

‍

Despite the seemingly large number of points reduced, the quality of the work remains virtually unchanged. At DataHunt, we're using this process to deliver better polygons to our workers.

Results of applying AI to image segmentation

To see how much the above process helped with the polygon work, we measured the efficiency of the work by directly comparing the time with and without AI in real-world tasks.

‍

The results showed a50% reduction in time on average, which means we were able to get more done in the same amount of time and spend more time on data quality.

Real applications of AI in image segmentation

These AI models are now being used in a current project that DataHunt and Samsung are working on together to produce even better results. Tens of thousands of images are being segmented using the AI, reducing work time and allowing for more thorough inspection of the work.

In this project, it was difficult to reduce the work time like the experiment above due to various criteria and other policies for each object, but there is a significant level of improvement and workers are satisfied with the use of AI.

Conclusion

In this article, we introduced image segmentation and what masks and polygons are, the process and real-world examples, and how we use AI at DataHunt to streamline our work, down to the quantitative level.

‍