A way of monitoring household appliances by using machine learning to analyse vibrations on a wall or ceiling has been developed by researchers in the US. Their system could be used to create centralized smart home systems without the need for individual sensors in each object. What is more, the technology could help track energy use, identify electrical faults and even remind people to empty the dishwasher.
“Recognizing home activities can help computers better understand human behaviours and needs, with the hope of developing a better human-machine interface,” says team member and information scientist Cheng Zhang of Cornell University.
The system, dubbed VibroSense, comprises two core parts: a laser Doppler vibrometer and a deep learning model, which is a type of machine learning system.
Wall or ceiling
The vibrometer detects tiny vibrations by measuring the distortion of a laser beam reflected from a surface. For single-story houses, VibroSense’s laser is targeted at a central interior wall within the home, whereas in two-story residences, the researchers aim it at a ceiling. The researchers’ prototype device is cable of detecting vibration velocities of up to 1.25 m/s with a resolution of around 0.2 µm/s.
The deep learning model learns how to identify individual appliances based on their characteristic noises and distinct paths that these vibrations follow through the building. Training the model involves using samples of individual vibration patterns for each target appliance. It takes around 1-2 h, depending on the number of vibration sources being monitored.
In tests undertaken across five households over two days, the VibroSense prototype could identify the activities of 18 different common household objects with a 96% accuracy rate – even when the appliances were spread across different rooms and floors. This included picking out the signals generated by an electric kettle, a range hood, a microwave, a washing machine, an exhaust fan and even a dripping tap. In addition, the device was capable of distinguishing between different stages of individual appliance use with an average accuracy of more than 97%.
Energy-saving advice
“Since our system can detect both the occurrence of an indoor event, as well as the time of an event, it could be used to estimate electricity and water-usage rates, and provide energy-saving advice for homeowners,” says Zhang.
In addition, VibroSense may even be able to help homeowners detect electrical faults before they become critical. “The theory is that if there is any abnormal condition to certain appliance, their vibration pattern would change, which VibroSense probably can pick up and detect with appropriate training,” Zhang explains.
In the future, Zhang believes that it may be possible to create a more advanced version of the machine learning system that uses vibration data for assorted appliances to accommodate the addition of new appliances to a home without needing additional training.
Privacy issues
The concept is not without some inherent issues, however – specifically as regards to the privacy implications of the device being able to “spy” on the activities going on in neighbouring flats, should it be used within an apartment complex. For this reason, the researchers say, the design is perhaps best suited for implementation within detached dwellings.
Commercial development of VibroSense, Zhang commented, “would definitely require collaboration between researches, industry practitioners and government – to make sure this was used for the right purposes.”
Michael Connelly, an engineer at Ireland’s University of Limerick comments, “The VibroSense technology has the potential to greatly enhance the understanding of the complex home environment and find applications in other areas of vibration measurement”. However, Connelly, who was not involved in the development of VibroSense adds, “laser vibrometers are expensive thereby it will be challenging to commercially exploit the VibroSense system, particularly as the deployment of the Internet of Things in the home increases.”
Spray-on antennas for the Internet of Things
This is an issue that the researchers are aware of. According to Zhang, the current version of VibroSense would cost around $1000–$2000. However, he adds that similar results may be possible using cheaper sensors such as accelerometers or geophones.
With their initial study complete, the researchers are now moving to explore the potential of using similar, minimally intrusive sensing devices to monitor human activities within broader settings – with the goal of using the data gathered to both enhance human–computer interactions and also to explore potential diagnostic medical applications.
The research is described in the Proceedings of the Association for Computing Machinery on Interactive, Mobile, Wearable and Ubiquitous Technologies.
