Screenshots of several speakers, described with an emotion

The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS)

A database for emotional auditory‑only, visual‑only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities
Screenshots of several speakers, described with an emotion
Screenshot: Celina von Eiff

JAVMEPS is an audiovisual (AV) database for emotional voice and dynamic face stimuli, with voices varying in emotional intensity. JAVMEPS includes 2256 stimulus files comprising

  • (A) recordings of 12 speakers, speaking four bisyllabic pseudowords with six naturalistic induced basic emotions plus neutral, in auditory-only, visual-only, and congruent AV conditions. It furthermore comprises
  • (B) caricatures (140%), original voices (100%), and anti-caricatures (60%) for happy, fearful, angry, sad, disgusted, and surprised voices for eight speakers and two pseudowords.
  • Crucially, JAVMEPS contains (C) precisely time-synchronized congruent and incongruent AV (and corresponding auditory-only) stimuli with two emotions (anger, surprise),
    • (C1) with original intensity (ten speakers, four pseudowords),
    • (C2) and with graded AV congruence (implemented via five voice morph levels, from caricatures to anti-caricatures; eight speakers, two pseudowords).

We collected classification data for Stimulus Set A from 22 normal-hearing listeners and four cochlear implant users, for two pseudowords, in auditory-only, visual-only, and AV conditions. Normal-hearing individuals showed good classification performance (McorrAV = .59 to .92), with classification rates in the auditory-only condition ≥ .38 correct (surprise: .67, anger: .51). Despite compromised vocal emotion perception, CI users performed above chance levels of .14 for auditory-only stimuli, with best rates for surprise (.31) and anger (.30).

We anticipate JAVMEPS to become a useful open resource for researchers into auditory emotion perception, especially when adaptive testing or calibration of task difficulty is desirable. With its time-synchronized congruent and incongruent stimuli, JAVMEPS can also contribute to filling a gap in research regarding dynamic audiovisual integration of emotion perception via behavioral or neurophysiological recordings.

For further information, please note the associated publication

  • von Eiff, C. I., Kauk, J., & Schweinberger, S. R. (2024). The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS): A database for emotional auditory-only, visual-only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities. Behavior Research Methods, 1-13.

Terms of Use

All stimulus files of JAVMEPS are freely available for the scientific community via the following link: https://osf.io/r3xqw/External link.

Please cite the following reference when you use JAVMEPS:

  • von Eiff, C. I., Kauk, J., & Schweinberger, S. R. (2024). The Jena Audiovisual Stimuli of Morphed Emotional Pseudospeech (JAVMEPS): A database for emotional auditory-only, visual-only, and congruent and incongruent audiovisual voice and dynamic face stimuli with varying voice intensities. Behavior Research Methods, 1-13.

Still-image frame examples from JAVMEPS showing a single speaker displaying different emotions

Screenshot: Celina von Eiff

Corresponding Developer

  • Julian Kauk

    Department for General Psychology and Cognitive Neuroscience, FSU Jena

    Contact via: julian.kauk@uni-jena.de

  • Stefan R. Schweinberger

    Department for General Psychology and Cognitive Neuroscience, FSU Jena, stefan.schweinberger@uni-jena.de