Audiovisual Perception of Emotion and Speech in Hearing Individuals and Cochlear Implant Users

This research is funded by a grant by the Deutsche Forschungsgemeinschaft (DFG, Grant Ref.: Schw 511/25-1, within the DFG SPP 2392 Visual Communication).

C.I vE. has been supported by the German National Academic Foundation (‘Studienstiftung des Deutschen Volkes’) from April 2019 until September 2022.

Description

This project aims to deepen our understanding of audiovisual perception in both hearing individuals and cochlear implant (CI) users, particularly focusing on the neurocognitive mechanisms underlying emotional communication. CIs are sensory prostheses designed to provide individuals with sensorineural deafness a modified sense of sound. While speech perception has been the main goal of CI technology – and has generally been treated as a benchmark for success of intervention, leading to greatly improved speech perception with CIs over the last few decades – emotion perception in CI users remains a widely underexplored research area. Importantly, the relevance of thoroughly studying emotion perception in CI users is underscored by consistent positive correlations between this perceptual ability and quality of life. In contrast, previous research has demonstrated only a relatively weak relationship between speech perception and perceived quality of life in CI users. Although voices are often produced by speakers who are visible during communication, and emotions are multimodal in nature, audiovisual integration of vocal and facial emotions in CI users has largely been overlooked in previous research.

Utilizing advanced methods such as parameter-specific voice morphing and synchronized audiovisual stimuli, we investigate in this research project how CI users integrate vocal and facial information to perceive emotions. Moreover, we explore how crossmodal plasticity in the brain, driven by visual inputs, might enhance CI users' multimodal perception. The project also examines the temporal dynamics of audiovisual integration in emotion perception, hypothesizing that visual information may offer a processing advantage, especially for CI users. Finally, we develop and evaluate perceptual training interventions aimed at improving CI users’ vocal emotion perception abilities, and, ultimately, their quality of life.

For further information, please visit: https://vicom.info/projects/audiovisual-perception-of-emotion-and-speech-in-hearing-individuals-and-cochlear-implant-users/

Resources

We developed JAVMEPS, an audiovisual database for emotional voice and dynamic face stimuli, with voices varying in emotional intensity, in the context of this project (von Eiff et al., 2024, Behavior Research Methods).