Provided/Doug James Lab

The sounds of a piggy bank being smashed include the initial fracture and the clattering of ceramic shards and coins hitting the floor. Cornell computer scientists are synthesizing these sounds to match computer-animated images.

A delicate wine glass shatters on the floor. A rock is thrown through a window. A child smashes his piggy bank. Dramatic moments like these in an animated movie or video game or some future virtual reality won't seem realistic unless the sound matches the action.

Cornell computer scientists are developing technology to synthesize the sounds that go with computer-animated images of brittle materials being smashed. Their methods look at the computer graphic model that underlies the animation, figure out how a corresponding real object would vibrate when fractured, and how that vibration would create sound.

For years, filmmakers have dubbed in recorded sound, but it is difficult to get it to match the action. And in a game or virtual reality, programmers can't know in advance just how hard or far a wine glass will fall.

"We'll compute motion and appearance and sound in an integrated way," explained Doug James, associate professor of computer science. "We won't just compute motion and appearance and have the sound as something you bolt on afterward."

James and graduate student Changxi Zheng will present their work at the SIGGRAPH 2010 conference in Los Angeles July 25-29. It follows previous work in which James and his students created sounds of dripping and splashing water and the clattering falls of such thin-walled objects as garbage cans and plastic bottles.

When a rigid object is struck or hits the floor, it can be deformed until the stress exceeds its strength, and then it shatters, releasing the energy stored by the deformation. Research shows that the sound comes mainly from the way all the little pieces vibrate just after the break, rather than from the whole object in the instant of fracture.

The computer calculates how each shard will vibrate when given the amount of energy stored by the deformation. The calculation takes into account how far the object was dropped or how hard it was thrown to determine the amount of energy available. It assumes that more fracture will occur in areas that have been strained the most, creating more, smaller pieces.

In most cases, the initial smash is followed by the scattering of debris on the floor. To accelerate computation of those sounds, the sound synthesis program can treat each irregularly shaped shard as an ellipsoid of similar size and shape. Then it draws on preloaded "soundbanks" -- not recorded sounds, but computer routines for calculating the vibration of ellipsoids of various sizes and materials.

To refine their procedures, the researchers smashed real objects and photographed them with high-speed, slow-motion cameras and recorded the sounds, then compared the actual sounds with their computed simulations. To demonstrate the results, they created videos of the smashing of a wine glass, a dinner plate, a glass table filled with dinnerware and a piggy bank full of coins. See http://www.cs.cornell.edu/projects/FractureSound/ for examples.

All of this is still an approximation of the real thing, James admitted, but it's a start, he said, and one that needs to be done now to make ready for the coming of new, more powerful systems.

"This is the first time anybody's ever built computer-synthesized models of these events with sound," he said. "Everything after this will be better. The future's going to be very different. Computers will be a thousand times as fast. It will be insane."

The sound simulation work is supported by a $1.2 million grant from the National Science Foundation.