To boost efficiency in typical households – where people forget to take wet clothes out of washing machines, retrieve hot food from microwaves and turn off dripping faucets – Cornell researchers have developed a single device that can track 17 types of appliances using vibrations.

The device, called VibroSense, uses lasers to capture subtle vibrations in walls, ceilings and floors, as well as a deep learning network that models the vibrometer’s data to create different signatures for each appliance – bringing researchers closer to a more efficient and integrated smart home.

“Recognizing home activities can help computers better understand human behaviors and needs, with the hope of developing a better human-machine interface,” said Cheng Zhang, assistant professor of information science and senior author of “VibroSense: Recognizing Home Activities by Deep Learning Subtle Vibrations on an Interior Surface of a House from a Single Point Using Laser Doppler Vibrometry.” The paper was published in Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies and will be presented at the ACM International Joint Conference on Pervasive and Ubiquitous Computing, which will be held virtually Sept. 12-17.

“In order to have a smart home at this point, you’d need each device to be smart, which is not realistic; or you’d need to install separate sensors on each device or in each area,” said Zhang, who directs Cornell’s SciFi Lab. “Our system is the first that can monitor devices across different floors, in different rooms, using one single device.”

In order to detect usage across an entire house, the researchers’ task was twofold: detect tiny vibrations using a laser Doppler vibrometer; and differentiate similar vibrations created by multiple devices by identifying the paths traveled by the vibrations from room to room.

The deep learning network was trained to distinguish different activities, partly by learning path signatures – the distinctive path vibrations followed through the house – as well as their distinct noises.

The device showed nearly 96% accuracy in identifying 17 different activities across five houses – including dripping faucets, an exhaust fan, an electric kettle, a refrigerator and a range hood – in five houses over two days, according to the paper. VibroSense could also distinguish five different stages of appliance usage with an average accuracy more than 97%.

In single-story houses, the laser was pointed at an interior wall at the center of the home. It was pointed at the ceiling in two-story homes.

The device is primarily useful in single-family houses, Zhang said, because in buildings it could pick up activities in neighboring apartments, presenting a potential privacy risk.

“It would definitely require collaboration between researches, industry practitioners and government to make sure this was used for the right purposes,” Zhang said.

Among other uses, the system could help homes monitor energy usage and potentially help reduce consumption.

“Since our system can detect both the occurrence of an indoor event, as well as the time of an event, it could be used to estimate electricity and water-usage rates, and provide energy-saving advice for homeowners,” Zhang said. “It could also prevent water and electrical waste, as well as electrical failures such as short circuits in home appliances.”

Co-first authors are visiting students Wei Sun, Tuochao Chen and Jiayi Zheng; Lucy Wang ’22; Matthew Dressa ’21; master’s student Benjamin Steeper ’19; Feng Tian of the Chinese Academy of Sciences; and visiting students Zhenyu Lei and Peng He also contributed.