Assessing Internal and External Attention in AR using Brain Computer Interfaces
Nataliya Kosmyna, Chi-Yun Hu*, Yujie Wang*, Qiuxuan Wu*, Cassandra Scheirer, Pattie Maes
Assessing Internal and External Attention in AR using Brain Computer Interfaces: A Pilot Study. 2021 IEEE 17th International Conference on Wearable and Implantable Body Sensor Networks (IEEE BSN '21), Best Poster Award, pp. 1-6, DOI:10.1109/BSN51625.2021.9507034.
A Pilot Study using Covert Visuospatial Attention as an EEG-based Brain Computer Interface to Enhance AR Interaction. In 2021 International Symposium on Wearable Computers (ISWC '21). Association for Computing Machinery, New York, NY, USA, 43–47. DOI:10.1145/3460421.3480420.
MIT Media Lab website link:
Keywords
BCI, EEG, AR, External/Internal Attention, Hololens2, HMD
Abstract
Most research works featuring AR and Brain-Computer Interface (BCI) systems are not taking advantage of the opportunities to integrate the two planes of data. Additionally, AR devices that use a Head-Mounted Display (HMD) face one major problem: constant closeness to a screen makes it hard to avoid distractions within the virtual environment. In this project, we reduced this distraction by including information about the current attentional state. We first introduce a clip-on solution for AR-BCI integration. A simple game was designed for the Microsoft HoloLens 2, which changed in real-time according to the user's state of attention measured via electroencephalography (EEG). The system only responded if the attentional orientation was classified as “external.” Fourteen users tested the attention-aware system; we show that the augmentation of the interface improved the usability of the system. We conclude that more systems would benefit from clearly visualizing the user's ongoing attentional state as well as further efficient integration of AR and BCI headsets.
Hololens 2 AR headset with embedded brain-sensing component
Figure 1. From left to right: hololens 2 AR headset with embedded brain sensing component (eeg clip-on holder over visual and pre-frontal cortex, two EEG electrodes highlighted in orange); A close-up image of the clip-on holder featuring 7 electrodes; user wearing an “enhanced” AR-BCI system (clip-on printed in blue; over visual and pre-frontal cortex); A visual rendering of the soccer ball being generated when the user is externally engaged in the game.
EEG clip-on hardware piece designed for hololens 2 headset
Figure 2. EEG clip-on hardware piece designed for hololens 2 headset. The clip-on piece includes three parts: the first two parts house EEG electrodes that make contact with the forehead (left image, highlighted in orange) and the back of the head (right image, highlighted in orange), while the last part, attached at the back of the hololens 2, holds the electronics (right image, highlighted in light green).
Questionnaires and Performance Assessment
Performance results of the attention-aware system for each participant in percentage. The accuracy is the average fold accuracy during the 5-fold cross-validation. The reported error is the percentage of internal trials in which at one point the balls mistakenly appeared. The questionnaire scores for each participant, mean and standard derivation.
MWQ = mind wandering questionnaire;
SUS-O = system usability score of the attention-unaware system;
SUS-N = system usability score of the attention-aware system;
DIS-O = distraction score of the attention-unaware system;
DIS-N = distraction score of the attention-aware system.
Context
The field of Brain-Computer Interfaces has seen strong advancement within the last 15 years. Many areas including rehabilitation, robotics, accessing the mental states of the user, and entertainment have used BCIs to research the intersection of wearable technology and physiological sensing.
However, the usability of these systems remains limited: they are bulky, tethered, expensive, uncomfortable, and prone to classification errors. Thus, many modern BCI systems are used in conjunction with other input modalities, like gaze trackers, virtual reality (VR) headsets and other HMDs, as well as augmented reality (AR) headsets, in order to compensate for the complexity of professional brain-sensing equipment.
We focus specifically on AR systems in this paper. BCIs and AR headsets are undeniably related. Both systems are designed to be worn on the head; HMDs often require hands-free interaction, which BCIs can provide; and HMDs often have voice and/or eye tracking features, which can mitigate the classification errors that arise during BCI use.
Contributions
In this paper, we integrated an EEG-BCI system with an AR headset, designed a simple 3D game and coupled the prototype with a real-time attention classifier. We were able to implement an end-to-end system and performed a user study with 14 participants. We also found a significant difference in usability ratings between the two systems, with better usability scores for the attention-aware system. Participants who achieved higher scores on the MWQ also rated the distraction of the two systems higher. Despite errors in the internal condition of the attention-aware system, it seemed that the users felt an improvement in the overall usability of the system and reported a decrease in perceived distraction.
Thus, we encourage more inquiry about the integration of AR and BCI systems to ensure that users are comfortable with using and wearing such a system for extended periods of time. This is crucial in order to be able to validate all the studies in reallife scenarios. Furthermore, though we have used an example of a simple game, our approach can be applied to video tutoring or training systems, where the user switches between taking in information and mentally processing it. Our future work will focus on implementing such scenarios and improving the classification accuracy of the system.
