Your Next Smart Selection Could Be Powered By A Neural Network

Admit it! object masking sucks. It’s not that it is impossible, even the hardest selections and masks can be created with some work (and some methods require less work that others). But, in general, masking is a hard and tedious work.

Researchers in the The Chinese University of Hong Kong working with Adobe Research are now showing some work that uses Convolutional Neural Networks to successfully mask portraits. The paper bears the boring name “Automatic Portrait Segmentation for Image Stylization” [PDF link], and it shows how selection and masking can significantly improve if the software knows that it’s making a portrait.

In an essence, the method shows how using several Neural Networks to successfully mask a portrait (and then suspect it to “stylising”). The article focuses on selfie shots (which would explain the stylising motivation), from what I gather, “stylising” is university-speak for instagram filters). In the researcher’s words:

The bulk of these portraits are captured by casual photographers who often lack the necessary skills to consistently take great portraits, or to successfully post-process them. Even with the plethora of easy-to-use automatic image filters that are amenable to novice. This work was done when Xiaoyong was an intern at Adobe Research. photographers, good portrait post-processing requires treating the subject separately from the background in order to make the subject stand out.

But, there is another reason, selfies are not taken by professional photographers (well, not always), so the image may not be well composted, it may not be in goo orientation, face parts may be hidden and other such shenanigans. Once the Network is given a subject though, it can detect those variations and mask out the face.

The following flowchart shows the flow of the algorithm: after a face is detected, it is treated for alignment and normalization and then the final neural network masks it.

Is the algorithm perfect? I guess that considering the fact that some of the researchers are from Adobe, I guess we will find out in one of the next Photoshop versions.

[The Chinese University of Hong Kong (pdf) via SolsticeRetouch]