MULTIMODAL EMOTION RECOGNITION WITH DEEP LEARNING AND FUSION MECHANISM

Abstract

The increasing interest in multimodal techniques for emotion recognition has led to the incorporation of both physiological and non-physiological data. This study aims to develop an advanced emotion recognition model that combines electroencephalography (EEG), video, and audio data using a hybrid fusion strategy to enhance system performance. We investigated various fusion methods, including diverse algorithmic strategies and intermediate attention, which significantly boosted emotion recognition accuracy. Our comprehensive audio-video-EEG model, equipped with intermediate attention and hybrid multimodal fusion techniques, was successfully tested on the RAVDESS dataset and further tailored to a new dataset at Nazarbayev University's Multimedia Lab, which includes video, audio speech, and EEG data. This thorough integration effectively tackled multimodal signal fusion challenges, improving both model sophistication and accuracy. Experimental findings indicate that integrating physiological and non-physiological data substantially enhances the accuracy of emotion detection systems. With our tailored dataset, the top-performing model achieved an accuracy of 68\% for 21-second inputs and 57\% for 3.6-second inputs across five categories. Despite challenges such as variable temporal frames and data segmentation, the results highlight the potent effectiveness of multimodal fusion in advancing emotion recognition. This research not only paves the way for future advancements but also emphasizes the need for ongoing optimization and further exploration of signal integration techniques.