AUGMENTING MULTI-SENSOR DATASET WITH PHYSIOLOGICAL DATA FOR HUMAN ACTIVITY RECOGNITION

Abstract

Human Activity Recognition (HAR) has gained significant relevance in healthcare monitoring, smart spaces, and human-computer interaction systems. Despite the fact that conventional HAR depends greatly on inertial measurement units (IMU), they are frequently faced with the inability to differentiate between activities having similar motion patterns but yielding dissimilar physiological responses. The thesis investigates the utilization of photoplethysmography (PPG) signals combined with IMU data to improve the performance of HAR via multi-sensor fusion. We introduce a novel CNN LSTM regression model capable of generating synthetic PPG data from IMU signals to enable the augmentation of available HAR datasets lacking physiological data. Using the augmented MMAct dataset, we evaluate various deep learning models—namely, CNN, LSTM, DNN, and CNN-LSTM models—on three fusion methods to determine the optimal approach for multi-sensor HAR. Our trials demonstrate that the inclusion of synthetic PPG data consistently improves classification performance across all architectures considered, with our best-performing CNN model achieving 82% accuracy and F-measure of 81%, surpassing existing benchmarks. The results demonstrate the usefulness of physiological context for activity recognition systems and present a general methodology for augmenting motion-based HAR datasets with synthetic physiological signals. This study helps toward building more robust and accurate HAR systems that can be utilized for a variety of applications, such as healthcare monitoring, fitness tracking, and smart environment.