01. PhD Thesis

Permanent URI for this collection

https://nur.nu.edu.kz/handle/123456789/8135

Browse

Browsing 01. PhD Thesis by Subject "Artificial Intelligence"

Now showing 1 - 1 of 1
  • Thumbnail Image
    ItemRestricted
    AUGMENTED REALITY-BASED HUMAN MEMORY ENHANCEMENT USING ARTIFICIAL INTELLIGENCE
    (Nazarbayev University, School of Engineering and Digital Sciences, 2023-11-23) Makhataeva, Zhanat
    This thesis presents a novel approach to augmenting human memory using emerging technologies of augmented reality (AR), computer vision (CV), and artificial intelligence (AI). The newly proposed system creates an external representation of object location memory for indoor environments (i.e., objects placed on the three floors of the building) to replace internal mental representations created in the human mind. The system has two main components - a wearable module (i.e., an AR headset) and a computing module (i.e., a laptop computer with an Ubuntu Operating System). I designed a first-person view (FPV) application running on the wearable module of the system to sense the environment, send the sensed data to the computing module of the system for processing, and then receive the processed data about the surrounding environment from the AI module of the system. Based on the received data, the application generates and updates the external representation of the objects in the environment and the user path and integrates the digital representation into the real environment as the three-dimensional (3D) virtual object in front of the user’s eyes. For understanding the user and object locations, the computing module of the system uses a CV-based camera technology localization framework indoors and a real-time pre-trained object detector. Based on the user and object position data, the AI module of the system performs the object-to-location binding. Then, it sends this data to the AR headset to construct the external synthetic representation of the objects placed in the indoor environment of the multi-storey building. To explore the usability of the proposed system, I designed an experimental study and invited 26 participants (i.e., 12 females and 14 males) from the community of Nazarbayev University (NU) to complete the experimental task. In the study, participants were involved in two activities that required spatial memory skills, such as memorization of the location of objects in the building and pointing on the 2D map of the building the location of the object that they learned after the previous memorization process. In the first activity, participants walked along the three floors of the building and tried to memorize the locations and labels of the objects placed along the corridors. In the second activity, participants were asked to point on the two-dimensional (2D) plans of the three floors of the building the locations of memorized objects from the first activity. Participants completed the experimental task two times during the study. There were two sessions in the study. Participants were divided into two groups. The first group of participants completed the experimental task with the assistance of the AR system during the first session and without the assistance of the system during the second session. The second group of participants completed two sessions of experiments in the reversed order, at the beginning without AR assistance and then later with the assistance of the AR system. During the study, I tried to evaluate the usability of the system. Also, I compared the cognitive load experienced by participants when they completed the experimental task under two conditions (i.e., with and without the assistance of the AR system). In addition, I studied performance variables such as error rate and task completion time during the map-pointing activity. The results of the experimental study conducted with the participation of 26 people from the research and faculty community of NU revealed that the proposed human memory augmentation system helped to reduce the cognitive load of people during tasks that required memorization of object locations in the indoor environment and map-pointing on the 2D map of the environment the locations of learned objects. The results recorded during the map-pointing activity showed that participants made 7.52 times fewer errors and spent less time on the computer-based test when they performed the activity with the assistance of the AR system. After completing the experimental task with the assistance of the AR system, participants were asked to fill in the System Usability Score (SUS) questionnaire. The results showed that participants rated the system usability with an average score of more than 80% for both activities. I used statistical tests to evaluate the significance of the reported results of the behavioral study. In addition, I investigated whether the gender of the participants affected the results of the map-pointing activity. In this regard, the statistical test results applied to the recordings of the error rate and task completion time in map-pointing activity revealed no significant difference between the results of the male and female participants in the study. Also, I performed correlation analysis to study the dependence between pairs of variables such as different dimensions of the mental workload (i.e., mental demand, effort, temporal demand, frustration, physical demand, and performance), quantitative performance variables (i.e., error rate and task completion time), and usability. Correlation analysis showed a weak correlation between the usability recording and the mental workload. Results of the behavioral study indicated the potential of AR and AI-enhanced systems to help people during cognitive tasks that require human object location memory. This thesis highlights that combined capabilities of AR and AI technologies may result in the development of a new generation of smart technology-based solutions to support people’s cognition through augmented visualizations and sensor-based perception of the environment.