MULTIMODAL PERFORMANCE ANALYSIS DURING JOB INTERVIEWS

Abstract

Emotion recognition based on multimodal data has become an important research topic with a wide range of applications, including online interviews. The study of respondents’ performance through the analysis of multiple modes of data is essential for a deep understanding of their emotions and communication patterns. To solve this problem, this thesis proposes a new method of analyzing multimodal interviews that uses deep learning techniques to extract meaningful information from various sources, such as video, audio, and textual data. The proposed approach uses late fusion to integrate information from different sources and generate an overall sum mary of the interviews. The effectiveness of the proposed method is evaluated on the whole MIT interview dataset, which includes 138 mock job interviews conducted with MIT undergraduates. The experimental results demonstrate that our framework can efficiently analyze multimodal data to produce promising results. The proposed approach identifies and captures critical aspects of communication, such as tone, facial expressions, and language use, which can provide valuable information to inter viewers to improve the overall interview process. This research has implications for improving understanding of communication patterns in various contexts, including job interviews, and may have practical applications in other fields