Adaptive bottleneck transformer for multimodal EEG, audio, and vision fusion

Abstract

Facial and speech expressions are primary cues for emotion recognition, while EEG provides a complementary neural perspective when external signals are ambiguous or absent. Although each modality contributes unique affective information, integrating such heterogeneous signals remains a major challenge in multimodal fusion research. To address this, the Adaptive Multimodal Bottleneck Transformer (AMBT) is introduced as a novel architecture, enabling efficient cross-modal interaction through adapters embedded within intermediate Transformer layers. These adapters 1) enhance stability by leveraging bottleneck tokens to prevent premature collapse, 2) enrich backbone representations while preserving unimodal capacity, 3) enable seamless integration across heterogeneous Transformer architectures, and 4) enable parameter-efficient training with fewer than 1% additional trainable parameters. AMBT was evaluated on three benchmark datasets: EAV (85.1%), CREMA-D (90.9%), and DEAP (98.7%), demonstrating competitive performance across all datasets. This results demonstrate the ability of AMBT to exploit complementary multimodal signals in a computationally efficient manner. © 2026 Elsevier Ltd. All rights are reserved, including those for text and data mining, AI training, and similar technologies.