Computers synthesize sounds to go with graphics

Computer-generated imagery usually relies on recorded sound to complete the illusion. Recordings can, however, limit the range of sounds you can produce, especially in future virtual reality environments where you can't always know ahead of time what the action will be.

Doug James, associate professor of computer science, develops computer algorithms to synthesize sound "on the fly" based on simulated physics models. His research group has already invented sound synthesis methods for splashing fluids, fracturing glass and other solids, and noisy thin shells such as sheet metal.

Now they have devised methods for synthesizing more realistic sounds of hard objects colliding and the roar of fire. They report their latest work at the 2011 ACM SIGGRAPH computer graphics conference in Vancouver, Canada, Aug. 7-11.

To synthesize collision sounds, the computer calculates the forces computer-generated objects would exert if they were real, how those forces would make the objects vibrate and how those vibrations transfer to the air to make sound.

Previous efforts often assumed that the contacting objects were rigid, but "in reality, there is no such thing as a rigid object," the researchers say. Objects vibrate when they collide, which can produce further chattering and squeaking sounds. Resolving all the frictional contact events between rapidly vibrating objects is computationally expensive. To speed things up, their algorithm simulates only the fraction of contacts and vibrations needed to synthesize the sound.

"If you set a box on a flat surface there are many contacts," James explained. "We identify a small set of physically faithful contacts, the simplest set that will get the job done." Similarly, the algorithm selects from many different ways the objects can vibrate, ignoring those that produce inaudible sounds.

Demonstrations include the sound of a ruler overhanging the edge of table and "buzzing" when plucked, pounding on a table to make dishes clatter and ring and the varied sounds of a Rube Goldberg machine that rolls marbles into a cup that moves a lever that pushes a bunny into a shopping cart that rolls downhill.

Fire is animated by mimicking the chemical reactions and fluid-like flow of burning gases. But flame sounds come from things that happen very rapidly in the expanding gases, and computer animators do not need to model those costly details to get good-looking flames.

"We can simulate the low-frequency sound for flame animations, but not all the fiery details," James explained, "so we rely on models based on real fire sounds to paint the fiery details onto the low-frequency sound." This keeps the cost down and has the added benefit of providing "style controls" based on what type of real flame sound is used as input data.

The researchers demonstrated with a fire-breathing dragon statue, a candle in the wind, a torch swinging through the air, a jet of flame injected into a small chamber and a burning brick. The last simulation was run with several variations of the sound-synthesis method, and the results compared with a high-speed video and sound recording of a real burning brick.

The flame sounds still lack some of the "whoosh" in the middle frequencies, the researchers noted, and the process is not yet ready for real-time applications, needing several hours to synthesize fire sound for a short video clip, but they say more powerful parallel hardware will help.