Green-screen keying method cuts time, boosts quality in film compositing

Filming an actor in front of a green screen and then superimposing the actor over another background is commonplace in feature film production, but getting rid of all traces of the green screen remains a chore. A new "keying" method devised by Disney Research and ETH Zurich, however, improves the results substantially.

What's more, these superior results can be achieved in about one-tenth the time it takes a professional compositor to perform this task manually.

"Keying is a crucial intermediate step in adding synthetic backgrounds and objects seamlessly into live-action shots," said Markus Gross, vice president at Disney Research. "It's also a major bottleneck in postproduction, so a keying method that is both fast and produces high-quality results will have a major impact on the production of feature films."

A report on the method, which is based on a novel algorithm that computes underlying colors and their mixing ratios, will be presented at the Association for Computing Machinery's (ACM) Conference on Computer Graphics and Interactive Techniques (SIGGRAPH 2017) next summer in Los Angeles and has been published in the August issue of ACM Transactions on Graphics journal (ACM TOG).

A number of commercial software tools are available to help a compositing artist perform keying, said Tunç Aydin, research scientist at Disney Research. But those tools have difficulty discerning the green screen background from the foreground in regions of the image where the colors of objects mix with each other, he noted. This mixing can occur because of motion blur, intricate object boundaries, such as hair, or colors cast onto an object from the green screen itself. Whatever the cause, these imperfections usually force the artist to painstakingly paint those regions manually.

The research team opted to create a color unmixing algorithm, which can be used to extract multiple color layers using a color model of the image, said Yagiz Aksoy, a Ph.D. student at Disney Research and ETH Zurich. A subset of these colors is then combined to produce the final keying result.

The team created a user interface that enables the artist to create a color model of the image simply by scribbling over each distinct-colored area of the image. The shape of the scribble doesn't matter; the program simply needs to identify each area of color so it can create a representation of all the colors in the scene. The algorithm computes the underlying colors and their mixing ratios for each area.

The researchers showed that the method would be applicable for keying scenes with non-green-screen backgrounds. Also, the ability to represent the image with multiple color layers, rather than just foreground and background layers as in conventional methods, creates opportunities for new applications, Aydin said. Colors of objects in a scene, for instance, could be changed.

In comprehensive comparisons with commercial keying tools and with other published methods, the researchers showed that the results for their method were superior in quality as it required only one-tenth of the manual editing time it would take a professional artist to process the same content.

Combining creativity and innovation, this research continues Disney's rich legacy of inventing new ways to tell great stories and leveraging technology required to build the future of entertainment.