Annotation of the results obtained in 2020
Through the analysis of previously recorded MEG data, we identified oscillatory phenomena allowing us to discriminate spontaneous eye fixations from those fixations used for intentionally sending commands to the computer. Theta range desynchronization was the most pronounced effect we discovered. It was observed approx. 200-250 ms after the fixation started and increased until the feedback was given. This reaction was specific for both voluntary and involuntary fixations, though it was significantly stronger in voluntary fixations. The reaction was statistically significant in most sensors and was focused in the parieto-occipital area. The second most important reaction was the synchronization in the alpha range observed in the interval of -100…200 ms. It was evident on a cluster of sensors in the frontocentral region of the right hemisphere. The results of MEG classification after removing frequencies above 7 Hz did not differ significantly from the broadband signal (0…45 Hz). It remains unclear, whether the neural network classifier uses induced or evoked MEG components (on the previous stage we showed that the MEG components, phase-locked to fixations, that discriminate voluntary and spontaneous fixations in a statistically significant way, generally represent slow signal amplitude changes). Nevertheless, the results we acquired allow us to elaborate the MEG preprocessing in brain-computer interfaces (Vasilyev et al., in prep.). The deconvolution-based method of signal component separation, originally designed for the analysis of EEG registered on the background of natural eye movements (Dimigen and Ehinger, 2000), was adapted for the analysis of MEG registered during the gaze-based control. The method allowed us to dispose of components related to eye movements and confirmed the existence of a slow MEG component related to voluntary eye dwell, evolving from approx. 200 ms after the first fixation began. We demonstrated the similarity of forms and topography of responses emerging after the initial choice and its affirmation by gazing on the designated region of the screen (Vasilyev et al., in prep). Another method was developed to clarify the interpretations of weights in LF-CNN convolutional neural network (Zubarev et al., 2019), as it was necessary to examine which sensors and moments of time influence the classification more than others. Using this method, we confirmed that no artifacts of oculographic origin contributed to the classification of voluntary and spontaneous fixations using MEG data. We also showed the possibility of improving the accuracy of classifying voluntary and spontaneous eye fixations in a gaze control task via training the classifier on the data acquired from repeated MEG sessions conducted on different days. When the convolutional neural network LF-CNN (Zubarev et al., 2019) was applied, the accuracy indicator of ROC AUC classification in the group of 5 participants increased from 0.64 (training on data from the same day when the data for testing sample was collected) to 0.72 (training on 3-4 extra sessions). Since these results were achieved on a small group (5 participants; the classification improved in 4), they can be presented only as preliminary, and yet it appears that the MEG signal in fact bears more useful information for the eye-brain-computer interfaces (EBCI) than it could be judged by looking at the results of classification we got earlier. These results demonstrate that training the classifier on recordings of extra sessions conducted before the online experiments might pose advantages for the EBCI online experiments. A detailed comparison of eye movement-related activity during spontaneous and intentional eye dwells showed that in case of voluntary eye dwells a correcting saccade shifts the gaze to the center of the target object (to foveate), after which the microsaccadic movements get inhibited, while in spontaneous fixations no shifts to the center occurs and thus low amplitude saccades emerge regularly. We intend to use these results to improve the classification of voluntary and involuntary eye fixations. We designed a method allowing us to analyze the BOLD-signal related to eye moving “events” in close temporal proximity. By applying it to the data from the fMRI experiment where the participants played a game using gaze-based control inside the tomograph, we showed that faster reactions can be registered in just 2-3 seconds. The analysis of phase-amplitude dependencies demonstrated that the choice conforming eye fixation had been the source of the strongest fMRI signal (akin to results of MEG analysis) and identified the brain regions displaying activation in gaze-based control tasks. The BOLD-signals related to the gaze-based control significantly varied in phase (time of their emergence and peaking relative to the event) in different regions of the brain. The phases of the responses varied both upward and downward relative to canonical hemodynamic function with delays ranging from -3 to +2 seconds. The phases were homogenous and representative of distinct topographical clusters. We proved that topographical clusters were not random, and their high specificity in generation of responses with predetermined phase was distinct from the canonical hemodynamic function. We showed the high ergonomics of a simple (i.e. not requiring speech recognition technologies) voice confirmation of screen object selection committed by gaze when it was “congruent” with the context of the primary task performed by the user (Zhao et al., 2020). However, if the confirmation was carried out using a partially irrelevant word (“you”), the results worsened (significantly with respect to convenience scores (p=0.0008) and not statistically significant with respect to the selection time) when compared to confirmation by uttering the number of the object (the task presupposed a consequent selection of objects in numerical order and in reverse order). Since the completely congruent confirmation of gaze-based selections is almost impossible to provide, these results affirm the high prospects of utilizing passive brain-computer interfaces for confirmation (in this instance the issue of congruency does not exist, because no further actions of the user are required), i.e., the technology to which our project is devoted. On this stage of the project we also completed the preparation of several papers dedicated to the results we acquired on previous stages: Dubynin et al. (under review), Ovchinnikova et al. (under review), Zhao et al. (under review). We had to reschedule our project plans, since it became apparent that our progress towards higher accuracy of classification was being made too slowly to achieve the desirable level of classification by the end of 2020, meaning any new experiments would have been futile. The main reason for our slow advancement was rooted in insufficiency of our knowledge of the human brain mechanisms supporting gaze-based interaction and also about specifics of the eye movement-related activity itself in such interaction. Unfortunately, little work has been dedicated to this topic (the only study directly addressing these questions was by Shishkin et al. (2016), carried out as part of our previous project supported by Russian Science Foundation). It was because of that, and also due to serious restrictions for the experiments with human participants during the COVID-19 pandemics, that we had to postpone the preplanned experimental studies and instead immerse ourselves into in-depth studies of the data collected in previously made experiments, including the development of new methods for data processing. The results obtained thanks to this research opened up more possibilities for creation of highly effective classifiers of eye fixations, that might be utilized not only in the context of eye-brain-computer interfaces (EBCI), but in other human-machine interaction systems as well.

 

Publications

---

Annotation of the results obtained in 2018
In 2018, special attention was paid to the adaptation of deep learning methods for the EEG and MEG based automatic recognition of eye gaze fixations used to give commands to the computer. In particular, artificial neural networks with deep architecture, as well as methods for tuning their hyperparameters, were adapted to the tasks of the eye-brain-computer interface (EBCI) and tested on the data recorded in the EBCI paradigm (MEG and partially EEG). In the analysis involving EEG data recorded in the EBCI paradigm we obtained with EEGNet (Lawhern et el., 2016, 2018), after hyperparameter optimization, ROC AUC of 0.79 ± 0.08 (M ± SD) on a test sample that was not used in training or optimization of hyperparameters, a 16% increase compared to ROC AUC for the previously used sLDA classifier on the same sample (13 subjects). This result showed the presence, in the EEG within gaze fixations used for control, a much larger amount of information useful for the classification than previous results showed. Moreover, it became evident that a much larger number of “control” fixations can be successfully classified; before this result was obtained, it seemed to be possible that in a large portion of such fixations the "control markers" are completely absent. The result also showed the importance of in-depth optimization of neural network hyperparameters, so this procedure will be used in further work on the project. It is essential that the improvements were achieved even without the use of MEG, which, firstly, can be of great practical importance, and secondly, complements the possibilities of improving the classification accuracy through the joint use of MEG and EEG, and through operant conditioning. For the first time, the registration of MEG and EEG, synchronized with the registration of eye movements, was carried out during involvement of the participants (n = 23) into a game tasks that required computer control using eye gaze fixations. For the first time, differences in MEG were identified in the presaccadic time interval followed by gaze fixations used for intentionally giving a command to a computer, compared to spontaneous gaze fixations. The differences were most pronounced over the frontal cortex, both in the medial and dorsolateral regions of the left hemisphere. The significance of this result is that, based on it, it can be possible to significantly improve the accuracy of classification in the EBCI. Moreover, very early detection of intention (even before the beginning of the ”controlling” gaze fixation) might become possible, opening a way to create an especially fast EBCI. In addition, a number of other works were carried out, primarily of a methodological nature, preparing the launch of an EBCI based on MEG + EEG in a near real time (online) mode, and also aimed at improving its performance.

 

Publications

---

Annotation of the results obtained in 2019
In 2019, we obtained a number of fundamentally new results related to the differences of the evoked brain magnetic response between spontaneous eye gaze fixations and the fixations used for gaze interaction, i.e. for sending commands to a computer using gaze. The brain response synchronized with the fixation onset was most pronounced in [100,300] ms after the onset and was localized mainly in the left hemisphere of the cerebral cortex. The maximum effect affected (i) the frontal sensors located above the posterior sections of the frontal lobe, corresponding to the location of the frontal oculomotor fields; (ii) parietal-temporal sensors above the supramarginal gyrus, which belongs to the brain system of voluntary attention control; (iii) to a lesser extent, the marginal occipital-temporal sensors above the occipital-temporal gyrus, where the zones of the ventral visual flow are localized. Activation of the anterior oculomotor fields and ventral parietal zones of the voluntary attention system was previously observed (in fMRI) under conditions requiring arbitrary inhibition of gaze movement to an external stimulus (Matsuda et al. 2004; Tu et al. 2006; Ettinger et al, 2008). We have shown, for the first time, that activation of the same zones of the cortex occurs when the gaze is intentionally stopped to control in the absence of any external signal. Moreover, the characteristic “neural fingerprint” of control fixation appears in the evoked magnetic response 150-250 ms after its start, and this allows us to count on its use for tuning and training classifiers for fixations in the eye-brain-computer interfaces (EBCIs). Importantly, the above effects were not associated either with corrective fixational eye movements or with the planning of a subsequent saccade. More broadly, these results for the first time provide a key to the mechanisms of the brain involved in providing intentional control of the gaze in a situation where this control is fully voluntary and is not a response to visual stimuli or a modulation of such responses. The effectiveness of statistical analysis of MEG data was significantly enhanced by the development of a new method for working with such data based on mixed linear models using a moving reference interval (baseline). We also found that the initial idea of the EBCI could be modified in certain ways, that is indicated by the high classification accuracy obtained on the MEG data when extending the interval of data analysis in the region after feedback in response to a gaze fixation (the group average ROC AUC was 0.91) and when averaging the values at the output of the EEG classifier by consecutive pursuits of intentional or spontaneous objects (ROC AUC 0.83 when averaging the tracking of ten objects). These modifications are of limited use because they allow to determine the type of gaze behavior with delay. However, they may be of interest for developing neuro-adaptive technologies based on them, where they could likely be used to assess the level of feedback expectation. At the stage of 2019, the project also examined a wide range of possibilities for improving the accuracy of classification in EBCI by tuning, including automatically, hyperparameters of artificial neural networks for classifying EEG and MEG signals. The choice of methods made at the previous stage was confirmed and refined. A number of additional experimental studies were also carried out, which made it possible to clarify other issues related to the development of the EBCI technology.

 

Publications

---