Engineering case study / 2024–2025
Multimodal emotion recognition
Studying emotion when speech and text are incomplete.
First author · Model design, implementation & experiments
Research question
Speech and text provide complementary cues about emotion, but a multimodal system cannot assume that every input is complete. This research studied emotion recognition through arousal and valence representations under incomplete-data conditions.
My contribution
I led model design, implementation, experiments, and analysis as first author. The work connected speech and text representations to the arousal–valence formulation and examined how the model behaved when the available information was incomplete.
The challenge and approach
A single categorical label compresses a complex emotional state. At the same time, combining modalities introduces a dependency on the availability of both signals. The research addressed these concerns together: arousal and valence provide a dimensional representation, while incomplete data remains part of the modeling problem.
The engineering focus was the relationship between representation and input conditions. The experiment design needed to examine the model under the conditions the research claimed to address, rather than treating missing information as an unrelated preprocessing issue.
Outcome and takeaway
This work led to a first-author journal article in the 2025 volume of the Journal of Society for e-Business Studies, alongside conference presentations on the topic.
The methodological takeaway is to define the input conditions before interpreting model quality. An emotion model's behavior depends both on its representation and on what evidence is available to it.