Using mobile devices to augment reality can enhance creative play and exploration

A child need not choose between the immersive, but often passive world of digital media or the physical interaction of real-world games and activities. Scientists at Disney Research say augmented reality with mobile devices serves as the perfect bridge between the two and can enhance creativity in the process.

To demonstrate their concept, called Augmented Creativity, Disney researchers have developed several prototype apps that add computer-generated animation, sound and visuals to such activities as coloring books, scavenger hunts and team games that require physical interaction.

"Exploring the world and new activities is what childhood is all about," said Markus Gross, vice president at Disney Research. "Our research brings the seamless fusion of the real and virtual world together with an intelligent and creative gameplay. We believe that these concepts offer exciting virtual enhancements over real-world interactions."

The researchers presented their Augmented Creativity concept and related apps at the Symposium on Mobile Graphics and Interactive Applications.

"We have been able to use the power and popularity of mobile devices to direct renewed emphasis on traditional activities of creative play," said Robert W. Sumner, principal research scientist. "One of our most compelling prototypes is an augmented-reality musical arrangement app that provides a tangible way for children to explore different music styles and arrangements by adding, removing, and re-arranging physical cards that represent different instruments and styles."

Another app, based on coloring, allows children to customize 3-D animated characters simply by coloring them as they normally would in a coloring book."

In a prototype for a multi-player game, players each use a mobile tablet to track a virtual object as they move around and talk with each other to cooperatively frustrate an invading alien force. They also developed a city-wide gaming framework, enabling the development of games such as scavenger hunts that get players outdoors, searching for interactive elements superimposed on buildings, parks and roads.

But Augmented Creativity isn't just about play. The researchers created a framework to help users write interactive narratives - a trickier task than standard storytelling because it requires the user to define and implement all possible interactions that someone might experience within the narrative. Another app helps children grasp the concept of programming robots; the child can program the robot to perform certain behaviors and then use augmented reality to see not just how the robot behaves, but exactly how the program is being executed.

"Augmented reality can help children develop a deeper understanding of programming by making the dynamics of program execution visible," Sumner said. "And it opens up new opportunities in gaming, by making it possible to create new classes of games that include user-programmed characters."

In addition to Gross and Sumner, the research team included Fabio Zünd of ETH Zurich, Mattia Ryffel, Stéphane Magnenat, Alessia Marra, Maurizio Nitti, Gioacchino Noris and Kenny Mitchell of Disney Research, and Mubbasir Kapadia of Rutgers University.