SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models

[Submitted on 24 Jun 2026]

View PDF HTML (experimental)

Abstract:As multimodal conversational systems increasingly engage in spoken interaction, their ability to navigate paralinguistic social cues has become a critical bottleneck for natural human-AI communication. However, existing evaluations of machine emotional intelligence assess reasoning exclusively through isolated text or passive acoustic perception, overlooking the complex cross-modal reasoning required for active, multi-turn dialogue. We introduce \textsc{SpeechEQ}, a comprehensive framework designed to evaluate the sociolinguistic reasoning of Speech-Language Models (SLMs). The framework includes a validated dataset of 2,265 dialogues across 15 Emotional Quotient (EQ) subscales grounded in EQ-i 2.0 theory, along with a multi-turn evaluation protocol measured by our proposed Spoken EQ (SEQ) score inspired by human EQ assessments. Experiments show limitations in how both existing Speech Emotion Recognition and end-to-end Speech-Language Models understand and apply paralinguistic cues through speech. While end-to-end architectures outperform cascaded systems, \textsc{SpeechEQ} reveals that current multimodal models remain bottlenecked by a text-reliant “modality shortcut,” an alignment-induced “safety trap,” and “contextual amnesia,” highlighting the barriers to truly emotionally aware AI. Our benchmark can be accessed at this https URL and demo page at this https URL

Submission history

From: Liang-Yuan Wu [view email]
[v1] Wed, 24 Jun 2026 16:03:38 UTC (2,786 KB)