3Qs: Facial recognition is the new fingerprint

Ear­lier this month, the FBI began rolling out a $1 bil­lion update to the national fin­ger­printing data­base. Facial-​​recognition sys­tems, DNA analysis, voice iden­ti­fi­ca­tion and iris scan­ning will all con­tribute to the government's arsenal of Next Gen­er­a­tion Iden­ti­fi­ca­tion (NGI) data. We asked Ray­mond Fu, a new assis­tant pro­fessor with joint appoint­ments in the Col­lege of Engi­neering and the Col­lege of Com­puter and Infor­ma­tion Sci­ence, to explain the sci­ence behind one of these new tech­nolo­gies: facial-​​recognition software.

How does facial recognition work, and where is the state of the art now?

Face-​​recognition research has been pop­ular for more than two decades. Great advances have been made from researchers from a broad com­mu­nity, such as bio­met­rics, com­puter vision and machine learning. The state-​​of-​​the-​​art tech­niques have been applied to real-​​world sys­tems for appli­ca­tions in sur­veil­lance, secu­rity and foren­sics. Face recog­ni­tion is a tech­nology that requires high accu­racy, espe­cially when secu­rity and foren­sics fac­tors are con­sid­ered. The cur­rent chal­lenges are scal­a­bility of data­bases; large vari­a­tion fac­tors in dif­ferent envi­ron­ments; aging, makeup and pose fac­tors of faces; and faces in social-​​media spaces.

Face-​​recognition sys­tems start with face detec­tion and tracking. Com­pu­ta­tional algo­rithms detect face posi­tions and poses in an image and then extract them for pro­cessing and analysis. During this pipeline, a couple of major chal­lenges create bot­tle­necks for the per­for­mance of real-​​world sys­tems. Facial expres­sions, aging and makeup are key vari­a­tions that cannot be easily removed. Tech­niques of 3-​​D mor­phable mod­eling and local fea­tures have been devel­oped to mit­i­gate such vari­a­tions. Lighting vari­a­tions can sig­nif­i­cantly affect the recog­ni­tion accu­racy espe­cially when a system is used out­side. Bench­mark data­bases have been col­lected from well-​​controlled lighting sources for devel­oping lighting insen­si­tive fea­ture extrac­tion and ana­lyt­ical mod­eling for such purposes.

The increasing acces­si­bility of the social-​​media space presents yet a new chal­lenge to devel­oping a large-​​scale iden­tity data­base. Con­fu­sion of sim­ilar appear­ances, over­load com­pu­ta­tions and mul­tiple data sources bring up uncer­tain­ties in modern face recog­ni­tion. Addi­tion­ally, new trends of soft-​​biometrics, big data and mul­ti­modality face recog­ni­tion have opened up new research thrusts.

What are the challenges and differences between identifying a single presented face and picking faces out of a crowd?

Face recog­ni­tion and iden­ti­fi­ca­tion are two dif­ferent prob­lems. Face recog­ni­tion is to match a person's face against a set of known faces and iden­tify who he or she is. For example, in a crim­inal inves­ti­ga­tion, a detec­tive may want to ID a sus­pect from a face image cap­tured on a sur­veil­lance camera.

Iden­ti­fi­ca­tion is to val­i­date the match of a given face and the claimed ID. For example, if an employee wants to access a secured area in a clas­si­fied depart­ment, she shows her ID card to the sensor while a camera cap­tures her face to match it with the record retrieved from the ID card input. If the match passes, the door will open automatically.

How would you address concerns raised by privacy advocates?

Face recog­ni­tion can be either pas­sive or active. In the air­port, for example, the sur­veil­lance cam­eras are taking videos in real time. Pas­sen­gers' faces are cap­tured in a pas­sive way. Online social-​​media spaces, like Face­book, pro­vide public domains for users to share their photos in an active way. Both may involve pri­vacy issues. How to bal­ance the pri­vacy issues and the public needs of secu­rity and human-​​computer inter­ac­tion are new research topics in this era.

In my research group, we have been funded by Air Force Office of Sci­en­tific Research, IC Postdoc Fel­low­ship and Google Research on these issues. Our research is mainly focused on under­standing social status and net­working of social-​​media users and their pri­vacy con­cerns. We are working on new com­pu­ta­tional method­olo­gies that could well ana­lyze the visual con­tent of social media and pro­vide auto­matic solu­tions for human-​​computer inter­ac­tion that could advance future social-​​network ecosystems.