Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using Electrophysiological and Kinematic Activity

Abstract

As virtual reality (VR) technology continues to gain prominence in commercial, educational, recreational and research applications, there is increasing interest in incorporating physiological sensors in VR devices for passive user-state monitoring to eventually increase the sense of immersion. By recording physiological signals such as the electroencephalogram (EEG), electromyography (EMG) or kinematic parameters during the use of a VR device, the user’s interactions in the virtual environment could be adapted in real time based on the user’s cognitive state. This dissertation evaluates the feasibility of passively monitoring cognitive workload via electrophysiological and kinematic activity while performing a classical n-back task in an interactive VR environment. The results indicate that scalp measurements of electrical activity and controller and headset tracking of kinematic activity can effectively discriminate three workload levels. Since motion and muscle tension can create co-varying task-related artifacts in EEG sensors mounted to the VR headset, decontamination algorithms were developed. The newly developed warp correlation filter (WCF) and linear regression denoising were applied on EEG, which could significantly decrease the influence of these artifacts. Analysis of the scalp recorded spectrum suggest two transient activity (termed pulse-decay effects) that impact feature extraction, modeling, and overall interpretation of workload estimation from scalp recordings. The best classification accuracy could be achieved by combining EMG, EEG and kinematic activity features using an artificial neural network (ANN)