New method captures facial details at high fidelity and real time

Forehead wrinkles that rapidly deepen and crow's feet that appear suddenly around the eyes might distress the average person, but the ability to quickly and realistically incorporate such details in a facial reconstruction is the key feature of a new performance capture method developed at Disney Research.

The method, which requires only a single video camera such as a webcam, is the first to both operate in real-time and to capture facial features in high resolution, including such details as wrinkles.

"This could open up a variety of new applications, from casual home users who wish to give their own likeness to a game character to commercial applications where, for example, a customer can apply and view virtual makeup," said Derek Bradley, research scientist at Disney Research. "In film production, our method would allow a director to assess an actor's performance transferred to a digital creature even as he is filming."

In addition to Bradley, the research team included Chen Cao and Kun Zhou of Zhejiang University as well as Thabo Beeler of Disney Research. The researchers will present their facial performance capture method at ACM SIGGRAPH 2015, the International Conference on Computer Graphics and Interactive Techniques, in Los Angeles Aug. 9-13.

Capturing facial details has proven critical for conveying emotions or the state of mind of a character and audiences are sensitive to subtle characteristics of the face. Animators thus devote great care to tracking facial performances of actors so that they can create realistic virtual characters. Until now, these facial performance capture methods have been designed to either capture these details in high resolution offline, or to record less detail but at high speed.

The new method represents a substantial step towards the goal of attaining both high quality and high speed.

As with other real-time facial capture methods, the researchers begin by using generic, low-resolution face models as the basis for reconstruction. A face model is virtually fitted to the user by the system as the user changes expressions and contorts his or her face in front of a video camera.

The system then automatically calculates a personalized wrinkle probability map, based on the set of extreme expressions the user assumed to train the program. The wrinkles may not match the user's actual wrinkles exactly, but they exhibit a very similar appearance because of the way they are formed.

"By learning the relationship between local image appearance and local wrinkle formation, we can reconstruct plausible face wrinkles in real-time from a single camera," Beeler noted. Once trained, the online capture approach can be applied to any new user without additional training. The system is generic and can be used with most common types of real-time face trackers.