MULTI-SENSOR FUSION FOR ROBUST FALL DETECTION AND CLASSIFICATION: A DEEP LEARNING APPROACH WITH MISSING MODALITY ADAPTATION

Abstract

Falls are a major concern for the well-being of elderly and disabled individuals. Timely detection of falls could play a crucial role in preventing the severe consequences of these accidents. This study proposes a deep learning-based multi-sensor fusion framework that integrates camera, wearable sensor data as well as other sensors. To allow a uniform CNN-based pipeline for both camera and sensor data, the methodology involves the conversion of 1D sensor data into 2D Recurrence Plot images. In total, four models that utilize either feature-level or input-level fusion strategies were trained and evaluated on the publicly available fall dataset. All the models were trained using a windowing approach with overlapping segments for potential real-world usability. Two of the models were trained following the multi-task learning approach, meaning apart from fusion heads, sensor-based independent training was employed. Results of the performed experiments reveal that A2 was a slightly better performing binary and multiclass classification model, with all the important metrics being above 99%. On the other hand, models that were implemented using the multi-task learning approach did not demonstrate a significantly higher resiliency to missing modality scenarios. This study was able to achieve robustness to missing data without significant performance sacrifices. The main aim of this study was to contribute to the development of sensor-agnostic networks that could also be potentially used in real-time scenarios.