Artificial intelligence (AI) is becoming better at understanding human behavior. One growing area of interest is whether machines can also recognize human emotions. This technology, often called emotion AI or affective computing, is already being used broadly in marketing, design, and even public services.
In this article, we explore how AI infers emotion from facial, vocal, and behavioral cues and the implications for future research.
A Short History of Emotion Recognition in AI
Researchers have long studied how humans express emotion through nonverbal cues.
One of the foundations of emotion recognition technology is the Facial Action Coding System (FACS), developed in the 1970s by psychologist Paul Ekman and W.V. Friesen. FACS identifies small muscle movements in the face called action units that can be used to describe expressions in an objective way. Originally designed for behavioral research, FACS was later updated and expanded, establishing itself as a widely used tool in the psychology field. Today, it serves as the basis for many AI systems designed to detect emotions by analyzing facial movements, though FACS itself does not label emotions.
In 1997, MIT professor Rosalind Picard introduced the concept of affective computing. This breakthrough was focused on giving machines the ability to detect emotional states and respond to them accordingly.
As machine learning and computer vision advanced (2000s–2020s), it became possible to train AI systems to detect emotions more accurately. Now, these tools can analyze facial expressions, vocal tone, and even behavioral patterns. Their applications have also expanded and are increasingly used in areas like media testing, virtual assistants, and user experience research.
And finally, a recent study published in 2025 in the journal Communications Psychology, found that AI models were better than humans at identifying emotions in written scenarios. We even have the test scores to prove it! AI scored 82% accuracy versus 56% for humans.
How Emotion AI Works Today
Modern AI can analyze different types of signals to estimate emotions. These include:
- Facial expressions: AI can detect subtle muscle movements to estimate whether someone looks happy, surprised, or confused.
- Voice tone: Speech patterns such as pitch, pace, or hesitation can be analyzed to identify signs of frustration, stress, or excitement.
- Behavior: AI can look at actions like repeated clicks or quick scrolls to deduce emotional states during a task.
- Multimodal analysis: Some systems combine visual, vocal, and behavioral data to improve accuracy.
Modern AI solutions use webcam recordings to track facial expressions during ad testing and analyze speech and expression to estimate emotions from images and texts in real time.
These tools are now used in areas like customer service, product testing, advertising, and online education:
- Ad testing: Researchers use facial recognition tools to measure emotional reactions to video content. For example, if viewers smile during a particular scene in a commercial, that moment may be evaluated as effective at eliciting an emotional response.
- Support calls: Voice analysis tools help detect signs of stress or frustration, so support agents can respond more effectively.
- Product and UX research: AI tools can track confusion or hesitation during product use, helping teams identify pain points.
- Education and healthcare: Emotion AI is being tested to track student engagement and detect early signs of depression.
While AI does not use empathetic feelers to pick up on emotions as humans might, it has become very effective at using patterns in data to make educated guesses about how someone might be feeling.
Challenges and Limitations
In its current state, AI models have the following challenges and limitations to remain aware of:
- Accuracy: People express emotions differently. A neutral face in one culture may appear an unhappy one in another. AI systems can misread these signals without sufficient context.
- Bias: Any type of AI model will reflect and share the biases of the data it was trained on. For instance, some models may be less accurate when analysing people with non-native accents, if the relevant data wasn’t included during training.
- Privacy: Emotion data is personal and participants of AI research must be informed and give consent. There are growing concerns about monitoring people’s emotions without their knowledge.
- Overinterpretation: A single smile or frown doesn’t always mean absolute happiness or sadness. Human emotions are complex and don’t always follow predictable patterns.
What This Means for Research
Emotion AI opens new possibilities for understanding how people respond, not just through what they say, but through how they react. It can capture immediate, nonverbal cues like facial expressions or vocal tone, offering valuable insight into emotional engagement, attention, or discomfort.
In fast-changing environments — such as during a crisis, a campaign launch, or a shifting policy debate, these tools can provide real-time signals before traditional methods can catch up. They help researchers identify when people react, how strongly, and to what.
With that said, emotion data alone has limitations. Facial expressions and voice tone don’t always reflect the full story or varying reasons for a given reaction. That’s why emotional AI works best when paired with clear, well-designed questions and unbiased data collection. Together, these methods can offer a more complete and accurate view of how people truly think and feel.
Interested in exploring how emotion measurement can enhance your research? Contact us at ask@riwi.com to learn how emotional insights can help you better understand and tailor communications to your audience.