High-tech sticker can identify real human emotions

UNIVERSITY PARK, Pa. — Saying one thing while feeling another is part of being human, but bottling up emotions can have serious psychological consequences like anxiety or panic attacks. To help health care providers tell the difference, a team led by scientists at Penn State has created a stretchable, rechargeable sticker that can detect real emotions — by measuring things like skin temperature and heart rate — even when users put on a brave face.

The researchers recently unveiled the wearable patch that can simultaneously and accurately track multiple emotional signals in a study published in the journal Nano Letters and filed a provisional patent for the technology. 

“This is a new and improved way to understand our emotions by looking at multiple body signals at once,” said Huanyu “Larry” Cheng, the James L. Henderson, Jr. Memorial Associate Professor of Engineering Science and Mechanics at Penn State and lead author on the paper. “Relying only on facial expressions to understand emotions can be misleading. People often don't visibly show how they truly feel, so that’s why we’re combining facial expression analysis with other important physiological signals, which will ultimately lead to better mental health monitoring and support.”

The sticker-like patch tracks a range of physiological responses, such as skin temperature, humidity, heart rate and blood oxygen levels, that are associated with emotional states. Most importantly, Cheng explained, the device's sensors are designed to work independently, minimizing any interference between the different measurements.

The device combines its analysis of the physiological signals with facial expression data to better distinguish between genuine emotions and acted ones. It then wirelessly transmits the real-time measured data to mobile devices and the cloud, where clinicians could potentially use it to better assess patients virtually. The device does not record personal information, only signals, Cheng explained, meaning personal privacy is protected through the device’s design.

“This technology has the potential to help people who are struggling with their mental health, but maybe aren’t being fully honest with others or even themselves about how much they are struggling,” said Yangbo Yuan, co-author on the paper and doctoral student at Penn State studying engineering science and mechanics.

Cheng explained that the collected data could also help bridge cultural or social gaps, which can manifest as a person appearing more stoic or expressive to their health care providers. “By keeping track of these signals, it could be possible to detect problems like anxiety or depression earlier in its progression.”

The researchers built the stretchy, BandAid-sized device by folding together thin layers of flexible metals like platinum and gold and cutting them into wave-like shapes that maintain sensitivity even when pulled or twisted. They also used layers of materials that change the flow of electrical current with temperature and built in hollow tubes made of carbon atoms, which absorb water and track humidity levels.

The multiple sensors were designed and placed on the device in such a way that they would not interfere with each other's measurements. For example, the researchers put a rigid layer under the temperature and humidity sensors to protect them from the stretching that the facial expression sensors would experience. They also used a waterproof layer to protect the temperature and strain sensors from humidity.

“We've engineered this device to measure these different signals independently, without them interfering with each other, providing a much clearer and more accurate picture of what's happening beneath the surface,” said Libo Gao, co-corresponding author on the paper and associate professor at Xiamen University.

Next the team trained an artificial intelligence (AI) model to read and understand signs of performed and real human emotion. The researchers recruited eight people, a common sample size for pilot studies, to perform six common facial expressions: happiness, surprise, fear, sadness, anger and disgust.

The participants displayed each emotional expression 100 times while the device tracked their movement. The researchers then fed this data to an AI model, training it to correlate specific facial movements with different emotions. They then recruited an additional three participants to further evaluate the model's abilities. It classified performed facial expressions with 96.28% accuracy.

When it came to tracking real emotions, the researchers tested how well the device tracked the psychological responses of the same participants as they watched video clips designed to elicit emotions. The device correctly identified emotions with 88.83% accuracy, with the sensors confirming that the psychological responses were consistent with known links between emotions and psychological reactions, such as increases in skin temperature and heart rate during surprise and anger.

Cheng noted that the ability to wirelessly transmit the data means that health care professionals could potentially monitor individuals remotely and provide timely emotional support through telemedicine.

“This sensor can serve a vital function in bridging gaps in access to care,” he said. “Given the rising stress levels in modern society, the ability to monitor emotions can provide early indicators of debilitating conditions and allow for proactive support.”

He explained that the device also opens the door for other systems for artificial intelligence (AI)-powered disease diagnostics and therapeutics beyond just emotion recognition. He noted there may be potential applications for clinicians to better understand the mental and emotional state of non-verbal patients, better identify behavioral and psychological symptoms of dementia and recognize opioid overdose. The technology could one day even be used for chronic wound monitoring and disease management, he added, as well as track neurodegenerative disease progression and athletic performance.

“While still in the research and development phase, this device is a significant step forward in our ability to monitor and understand human emotions, potentially paving the way for more proactive and personalized approaches to mental health care,” Cheng said.

Other contributors include Hongcheng Xu of Xi’an Jiaotong University. The U.S. National Institutes of Health and the U.S. National Science Foundation funded the Penn State researchers’ contributions to this work.