Researchers develop fast, economical method for high-definition video compositing

Video compositing to create special effects, replace backgrounds or combine multiple takes of an actor's performance is an integral, but highly labor-intensive, part of modern film making. Researchers at Disney Research, Zürich, however, have found an innovative way to create these composite videos that is simple, fast, and easy to use.

Rather than perform a painstaking segmentation of elements that are to be added or subtracted from a video, the Disney system, called DuctTake, uses computer algorithms to find a spatiotemporal "seam" through the video frame that enables two or more videos to be joined together.

These seams can be highly irregular, following the contours of people, furniture and other objects that are common in each take of the scene. Because it can only combine scenes that have overlapping content, DuctTake isn't useful for combining arbitrary video clips. But it works like a charm when combining multiple takes of the same shot.

The Disney Research, Zürich researchers showed the technique can be used across a wide range of video composites. For example, it can combine the best performances by actors from several different takes into a single seamless output, reducing the number of on-set takes that are required to be filmed. Furthermore, the same technique can be used to make a cut seamless, allowing the first half of one take to be combined into a second take. In another case, the researchers eliminated unwanted cars from a street scene by offsetting the vehicle with empty street imagery from the same video take. Hard-to-shoot scenes involving animals, they demonstrated, become a bit easier when an animal trainer can be easily removed from the shot, or other actors can be added at the moment of time when the animal does the right thing.

"The most delicate component is alignment," said Oliver Wang, post doctoral researcher at Disney Research, Zürich. "But given properly aligned views, we can almost always generate good composites with minimal work."

The findings by Wang and his Disney Research, Zürich team of Jan Rüegg, Aljoscha Smolic and Markus Gross were presented at Eurographics 2013, the European Association for Computer Graphics conference.

Most video compositing is accomplished now by the digital equivalent of "cut-and-paste." Rotoscoping is the process by which elements can be added by drawing segmentation outlines. Chroma-keying, familiar to viewers of TV weathercasts in which news announcers appear to stand in front of large, animated maps, separates actors from backgrounds based on color hues; it's cheap and robust, but restricts filming to studio environments, and can require challenging color balancing in post-production.

"Our approach solves a simpler problem," Gross acknowledged, "but as a result it is robust, fast to compute and easy for artists to use, enabling compositing techniques to be used on lower budget shots and productions."

A DuctTake user can combine two videos by making a few quick brush strokes to indicate which parts of the video to keep in each take. An algorithm developed by the Disney Research, Zürich team then computes an optimal seam and merges the two videos together.

DuctTake also includes a number of tools necessary to create a composite that looks realistic, such as adjusting the seam between frames to compensate for camera movement or content movement. Other tools adjust for differences in brightness, contrast and hue between takes, blend images along seams that are visible in a common background, and increase the blurriness in some video to match blurring that occurs in the video with which it is being combined.