1.

Introduction

There are numerous individuals with physical disabilities, for whom a variety of human–machine interface (HCI) systems have been developed. These systems are based on a variety of electrical or nonelectrical biosignals, such as electromyograms (EMGs),^1,2 electrooculograms (EOGs),^3,4 sip-and-puff signals,⁵ head movements,⁶ and tongue movements.⁷ Disabled individuals who still have the ability to move specific parts of their bodies can use these types of HCI systems, but these systems cannot be applied to those who are unable to move due to locked-in syndrome (LIS). Brain-computer interface (BCI) technology can help the patients with LIS to communicate with the outside world using their brain activity.

BCI is a technology that provides a direct communication pathway between the brain and external devices without the need for any muscular movements.⁸ In general, a BCI system can be implemented based on two different approaches: an invasive method or a noninvasive method. An invasive BCI system uses a bundle of microelectrodes directly implanted into the brain to record the neuronal-spiking activities. The invasive BCI enables high-precision control of external devices due to its high-quality brain signals, whereas the biocompatibility of the implanted microelectrodes and high risk of surgery are still crucial issues to be solved.^9–11 Currently, the noninvasive BCI approach has been studied more actively than the invasive BCI methods.¹² To implement noninvasive BCI systems, various brain signal recording modalities have been used such as electroencephalography (EEG),^13–15 NIRS,^16–18 functional magnetic resonance imaging (fMRI),^19,20 and magnetoencephalography (MEG).²¹ Among these, EEG has been most widely used due to its high temporal resolution, reasonable hardware price, and portability.^12,22,23

Recently, there has been growing interest in the NIRS-based BCI systems because NIRS is generally less susceptible to gross electrophysiological artifacts caused by eye blinks, eyeball movements, and muscle activity.¹⁶ So far, various mental imagery tasks have been used for NIRS-based BCI studies, such as hand motor imagery,^18,24 mental arithmetic tasks,^16,17,25,26 music imagery,^16,26 object rotation,²⁵ and letter padding.²⁵ Most of these NIRS-based BCI studies attempted to classify a pair of mental tasks with the goal of achieving the highest possible classification accuracy for the binary communication of patients with LIS.

To increase the BCI performance, previous NIRS-based BCI studies have developed advanced signal processing techniques and machine learning methods.^{16–18,24–27} However, some subjects did not show a classification accuracy high enough to be used for practical binary communications (less than 70%) because they failed to produce the distinct and consistent brain activity patterns expected when performing given mental tasks. Previous BCI studies have reported that this so-called “BCI illiteracy” phenomenon occurred in approximately 15% to 30% of the individuals who participated in various types of BCI experiments.^28–31

One of the solutions to circumventing the BCI illiteracy issue would be using individualized or customized mental task combinations instead of using a fixed set of mental tasks. However, it is difficult to select an optimal combination of mental tasks that can elicit distinct brain activity patterns and guarantee the best classification accuracy, because testing a large number of mental tasks for each individual would be a long and tedious preliminary process. In our experience, testing more than eight mental tasks in a single experiment is not efficient, because participants generally lose their focus due to mental fatigue. Therefore, studies investigating the intrinsic characteristics of mental tasks are necessary in order to more efficiently search for optimal mental task combinations for each individual. For example, mental task pairs showing a low classification accuracy can be excluded from the candidate mental task set, or vice versa. Since the number of available mental tasks that have been used in the BCI literature is very large (over 20), it is highly desirable to accumulate a series of references reporting the results of preliminary test experiments performed with different candidate mental tasks. However, no previous NIRS-based BCI studies have investigated optimal combinations of mental tasks with the aim of determining the best task combination among all possible pairs of various mental tasks in terms of classification accuracy.

The goal of this study was to investigate whether there are any combinations of mental tasks that are relatively more advantageous in increasing the classification performance of NIRS-based BCI systems. To compare the classification accuracies of various mental task combinations, we measured task-related concentration changes of oxygenated, deoxygenated, and total hemoglobin ([oxy-Hb], [deoxy-Hb], and [total-Hb]), while seven participants were carrying out eight different mental tasks. For the binary classification, we constructed four different feature sets for [oxy-Hb], [deoxy-Hb], [total-Hb], and a combination of [oxy-Hb] and [deoxy-Hb], and then a linear discriminant analysis (LDA) algorithm was applied to the features picked up by the Fisher criterion. The average classification accuracy for each of all possible mental task combinations (${{}_{8}C}_2=28$) was estimated by a $10\times 10\text{-}\text{fold}$ cross-validation.

2.

Methods

2.3.

Experimental Paradigm

Figure 1 shows the overall experimental paradigm used in this study. Before each session, a preparation time was given for 10 s, during which the participants waited for an upcoming instruction without making any movements. At the beginning of each trial, an instruction indicating one of the eight mental tasks was randomly presented for 5 s, during which the participants had to prepare for the mental task to be performed. For the SUB, MUL, and WRT tasks, the participants were asked to memorize a pair of two numbers, two-digit numbers, and four characters, respectively. A pure-tone beeping sound was presented for 125 ms, and then a fixation cross appeared at the center of the monitor for 15 s, which was the signal to start performing the designated mental task. After the participant performed the given mental task for 15 s, an empty screen was presented for a variable duration from 10 s to 15 s. This procedure was repeated twice for each mental task in one session. A total of 10 sessions were conducted, and thus each participant carried out 20 trials for each mental task.

Fig. 1

A schematic diagram describing the experimental paradigm.

2.4.

Near-Infrared Spectroscopy Data Recording

For the data recording, we used a multichannel NIRS imaging system (FOIRE-3000, Shimadzu Co. Ltd., Kyoto, Japan). Figure 2 shows the optode configuration used in this study. As shown in Fig. 2, 16 sources with wavelengths of 780, 805, and 830 nm, and 15 detectors were attached to each participant’s scalp. The center optode was placed on Cz according to the international 10–20 system (a standard electrode placement rule for EEG), and the inter-optode distance was set to 3 cm based on the previous NIRS studies reporting that the 3-cm inter-optode distance is ideal for measuring cortical hemodynamic responses.^16,26,33 The task-related concentration changes of hemoglobin were recorded at 50 different scalp locations with a sampling rate of 10 Hz.

Fig. 2

The configuration of sources, detectors, and NIR channels used in this study. The red and blue circles indicate the 16 sources and 15 detectors, respectively. The gray numbered squares represent the 50 channels. The distance between adjacent optodes was set to 3 cm.

2.5.

Near-Infrared Spectroscopy Data Analysis

2.5.1.

Preprocessing

The recorded raw light intensities of the three wavelengths (780, 805, and 830 nm) were converted into the concentration changes of oxygenated, deoxygenated, and total hemoglobin ([oxy-Hb], [deoxy-Hb], and [total-Hb]), using the modified Beer-Lambert law. Optical signals measured on the scalp are generally accompanied by several physiological signals that are not directly related to the cognitive activities.^34–38 The most dominant component is heart rate (arterial pulsation) showing a fundamental spectral peak around 1.4–1.8 Hz.^37,39 Breathing and Mayer waves are two other physiological components observed at approximately 0.3 and 0.1 Hz frequencies, respectively.^37–39 To remove these spontaneous activities from the recorded NIRS signals, a zero-phase low-pass filter with a cutoff frequency of $f_c=0.09\mathrm{Hz}$ (fourth-order Butterworth) was applied to each hemoglobin response. In addition, a zero-phase high-pass filter with a cutoff frequency of $f_c=0.01$ (fourth-order Butterworth) was used to remove the low-frequency baseline drifts. The frequency band of 0.01 to 0.09 Hz (or 0.1 Hz) has been widely used for filtering out the spontaneously generated physiological components from the NIRS signals. ^{16,27,39–42} Note that the filtering process was applied to each session of which the total duration was about 530 s, before segmenting the NIRS data.

2.5.2.

Feature extraction

Various types of BCI features have been used to classify hemodynamic responses to different mental states, such as amplitude,^{17,18,24,26,27,43–45} slope,^16,27,42 variance,^24,43 skewness,^24,43 kurtosis,^24,43 root mean square,⁴³ zero crossings,⁴³ wavelet coefficients,²⁵ and laterality.²⁷ Some studies used raw light intensity signals directly, without transforming them into concentration values of hemoglobin.^16,26,45 In this study, we used the mean hemoglobin concentration values over predefined time periods as candidate features, because they have been most widely used for NIRS-based BCI studies.^{17,18,24,27,43,44} To extract the task-related NIRS features, we used the 15-s epoch recorded while the participants were carrying out each mental task (see Fig. 1). The hemodynamic responses to mental activity are generally observed with a time delay of 5–8 s,^16,24,46 but this varies among individuals and mental task types. Therefore, to capture the task-specific hemodynamic responses more accurately, we used three different window sizes of 5, 10, and 15 s, and consequently tested six different time windows, i.e., 0–5, 0–10, 0–15, 5–10, 5–15, and 10–15 s, to extract the candidate features. The features were extracted by simply averaging the hemoglobin responses in each time window, and four different candidate feature sets were constructed for [oxy-Hb], [deoxy-Hb], [total-Hb], and a combination of [oxy-Hb] and [deoxy-Hb]. The combination of [oxy-Hb] and [deoxy-Hb] feature set was constructed by simply putting together the [oxy-Hb] and [deoxy-Hb] feature sets into a single feature set. Since six features were extracted for each of the 50 NIRS channels, the total numbers of the extracted features were 300 for the [oxy-Hb], [deoxy-Hb], and [total-Hb] feature sets ($6\text{features}\times 50\text{channels}$), and 600 for the combination of [oxy-Hb] and [deoxy-Hb] feature set ($6\text{features}\times 50\text{channels}\times 2\text{signal types}$ ([oxy-Hb] and [deoxy-Hb])).

2.5.3.

Feature selection

It is well known that an excessive number of features can cause not only the overfitting of the training data, but also an increase in the learning time of a pattern classifier due to irrelevant or redundant features contained in the high-dimensional feature vector. This will eventually degrade the overall classification performance of the trained classifier. Therefore, feature selection is an indispensable step in every classification problem. In order to reduce the dimensionality of the feature vector as well as to select the best feature subset, we used the Fisher score, one of the most widely used feature selection methods that has been successfully applied to previous NIRS-based BCI studies.^16,27,42 The Fisher score was estimated for each element of the constructed feature vector using

Eq. (1)

$${\mathrm{FS}}_k=\frac{{({\mu }_i-{\mu }_j)}^2}{s_i^2+s_j^2},$$

where $\mu$ and $s^2$ represent the mean and variance, respectively, and the subscripts $i$ and $j$ represent two different classes. The subscript $k$ designates the $k$’th feature element. A higher Fisher score implies that the distance between features in different classes is larger and the variance between features in the same class is smaller. Thus, the top $N$ features with the highest Fisher scores are generally selected for the classification. In this study, we selected the top five features for classification based on a previous NIRS-based BCI study, which reported that classification accuracy is no longer increased when the number of features selected by the Fisher criterion is five or six.¹⁶

2.5.4.

Classification

To prevent potential biases in estimating classification accuracy, we used a $10\times 10\text{-}\text{fold}$ cross-validation with an LDA classifier that has been most widely used in NIRS-based BCI studies.^{16,17,24,25,27,42,43,45} The whole dataset (20 trials for each class) was randomly split into 10 subsets each with the same number of trials (two trials per class). Nine subsets were used to train an LDA classifier, and the other subset was used for the estimation of classification accuracy. This procedure was repeated until every subset was tested, and the average classification accuracy was then evaluated. This “ten-fold cross-validation” was repeated 10 times with reshuffled subsets. In each cross-validation step, the top five features were selected independently, using the Fisher scores as described in the previous section. Since the $10\times 10\text{-}\text{fold}$ cross-validation was independently applied to each subject, the top five features varied among subjects. The classification accuracy was estimated for all 28 possible combinations of two mental tasks using the following four feature sets: [oxy-Hb], [deoxy-Hb], [total-Hb], and a combination of [oxy-Hb] and [deoxy-Hb]. Figure 3 illustrates an intuitive example of the classification process, where the best two features were used to classify two different mental tasks (MUL versus ROT).

Fig. 3

An example of a cross-validation step, when the MUL and ROT tasks were classified using the [oxy-Hb] feature set of the participant P6. (a) Feature values from 20 trials are displayed for each class on a two-dimensional feature space. (b) A linear discriminant analysis (LDA) classifier (green dotted line) is constructed using 18 trials of each class (red and blue circles), and the other two trials of each class (gray circles) are used for the evaluation of accuracy. (c) The remaining two trials of each class (red and blue circles) are then classified by the constructed LDA classifier.

3.

Results

Since classification accuracy should be greater than 70% for a binary BCI system to be used for practical communication purposes,⁴⁷ only mental task combinations with a classification accuracy over 70% were regarded as meaningful task combinations in this study.

Figure 4 summarizes the mental task classification results of each participant for four feature set types: ([oxy-Hb], [deoxy-Hb], [total-Hb], and a combination of [oxy-Hb] and [deoxy-Hb]), in which a rectangle filled with black color implies that the classification accuracy of the corresponding pair of mental tasks exceeded 70%. The “meaningful” mental task combinations varied among participants as well as varying with the feature sets used, but the combination of RMI and MUL tasks and that of MUL and ROT tasks showed classification accuracies of over 70% in most participants regardless of the feature sets.

Fig. 4

Mental task classification results of each participant for four different feature set types, where a filled rectangle implies that the classification accuracy of the corresponding pair of mental tasks exceeded 70%.

Figure 5 shows the mean classification accuracies of all possible mental task combinations, those of RMI and MUL combinations and those of MUL and ROT combinations, with respect to different feature sets. The classification accuracies averaged over all 28 possible combinations of mental tasks were slightly over the level of random chance ($=50\%$). However, the combination of RMI and MUL tasks and that of MUL and ROT tasks showed mean classification accuracies around 70% for most feature sets. Specifically, the combination of MUL and ROT tasks showed mean classification accuracies of over 70% in three of four feature sets (70.57% for the [oxy-Hb] feature set, 71.53% for the [deoxy-Hb] feature set, and 74.39% for the combination of [oxy-Hb] and [deoxy-Hb] feature set). The combination of RMI and MUL tasks showed a mean accuracy over 70% when the [oxy-Hb] feature set was used (70.1%). Note that the above two mental task combinations (RMI versus MUL, and MUL versus ROT) always ranked first or second in the classification accuracy among all 28 task combinations regardless of the feature set types.

Fig. 5

Mean classification accuracies of all possible combinations of mental tasks (denoted by “All”), those of the RMI and MUL combinations (denoted by “RMI vs. MUL”), and those of the MUL and ROT combinations (denoted by “MUL vs. ROT”) for different feature sets.

To explore further which mental task was most frequently selected in the mental task combinations with a classification accuracy over 70%, we counted the number of times that each mental task was included in the “meaningful” task combinations shown in Fig. 4. As shown in Fig. 6, three mental tasks, RMI, MUL, and ROT, were most frequently included in the meaningful mental task combinations. This result is in line with the previous results that the combination of RMI and MUL tasks and that of MUL and ROT tasks resulted in a higher classification accuracy than the other mental task combinations. These results suggest that the three selected mental tasks, RMI, MUL, and ROT, have the potential to yield higher classification accuracy than the other mental tasks.

Fig. 6

The number of times that each mental task was included in the mental task combinations with a classification accuracy of over 70%.

Figures 7(a) and 7(b) show the hemodynamic responses averaged across all subjects at the most frequently selected channels in the cross-validation processes, for RMI and MUL tasks and MUL and ROT tasks, respectively. The hemodynamic responses elicited by different mental tasks were significantly different at the selected channels, thereby making it possible to discriminate two different mental tasks with reasonable accuracy. Moreover, small standard errors computed across all subjects for each hemodynamic response indirectly show the high reliability of our experimental results.

Fig. 7

Grand-averaged [oxy-Hb], [deoxy-Hb], and [total-Hb] responses recorded during (a) RMI and MUL tasks and (b) MUL and ROT tasks. The shaded regions indicate standard errors computed across all subjects for each hemoglobin response.

Figures 8 and 9 show the grand-averaged [oxy-Hb] and [deoxy-Hb] responses, respectively, for three different mental tasks (RMI, MUL, and ROT tasks). The [oxy-Hb] responses acquired during RMI and ROT tasks showed similar patterns in frontal areas (ch. 37–ch. 50), but the difference between the two task conditions was increased in parieto-occipital areas. The [oxy-Hb] responses during the ROT task became more similar to those during the MUL task around the posterior areas. On the other hand, the [deoxy-Hb] responses of three mental tasks did not show any consistent spatial patterns.

Fig. 8

Grand-averaged [oxy-Hb] responses recorded during RMI, MUL, and ROT tasks for all channels.

Fig. 9

Grand-averaged [deoxy-Hb] responses recorded during RMI, MUL, and ROT tasks for all channels.

Since we used a relatively small number of trials for each mental task ($n=20$) and large numbers of features ($>300$), there is a possibility that high classification accuracy ($>70\%$) in some mental task combinations might arise by chance. To verify this, we newly generated eight datasets each consisting of 20 trials by randomly shuffling trials in the original datasets (i.e., exchanging trials between different classes), and then we repeated the same classification procedure ($10\times 10\text{-}\text{fold}$ cross-validation) with the newly generated datasets. As a result, the number of meaningful combinations of mental tasks (classification accuracy $>70\%$) was significantly decreased from 84 (10.7% of all mental task combinations) to 7 (0.9% of all mental task combinations) when the newly generated datasets were used for the validation. Our simulation results showed that about 8% of the mental task combinations with high-classification accuracy in the original validation results could arise by chance, but it is expected that such a small portion (8%) might not affect the overall tendency of our results and our conclusion. In Fig. 4, two selected mental task combinations, combination of RMI and MUL tasks and that of MUL and ROT tasks, showed the meaningful classification accuracy in 31 cases, which is about 37% of all meaningful combinations ($n=84$) and is overcoming the others in numbers. Therefore, even when we assume that 2 or 3 cases were selected by chance (with the 8% probability) in these two task combinations, the overall tendency would not be affected at all.

4.

Discussion

In order to implement a high-performance NIRS-based BCI system, different mental states should be discriminated with a high-classification accuracy. Most previous NIRS-based BCI studies have mainly focused on enhancing classification accuracy with the state-of-the-art signal processing methods and machine learning algorithms with the aim to increase the overall performance of a BCI system.^{16–18,24–26} However, if a BCI user cannot generate distinct brain signals related to certain mental tasks, even the most advanced methods might not be able to classify those mental tasks. In the present study, in order to provide a useful reference for the selection of optimal mental task combinations, we investigated the suitability of a variety of mental task combinations for BCI based on mental imagery tasks. To the best of our knowledge, the present study is the first that investigates which mental task combinations can be better choices in designing individualized NIRS-based BCI systems.

From the analysis results, we confirmed that the combination of MUL (mental multiplication) and ROT (mental figure rotation) tasks and that of RMI (right-hand motor imagery) and MUL tasks might be the most promising mental task combinations among the 28 combinations tested in this study. In particular, the mental task MUL was most frequently observed in the meaningful combinations of mental tasks ($>70\%$), suggesting that the MUL task generates a brain activity pattern most distinguishable from the other mental tasks. It is also noteworthy that the combination of MUL and ROT tasks was the only mental task combination with which all participants showed classification accuracies over 70%. In the case of the combination of RMI and MUL tasks, unfortunately, one participant (P1) did not show the meaningful classification accuracy ($>70\%$) for any feature sets. The combination of MUL and ROT tasks and that of RMI and MUL tasks showed relatively high classification accuracy for most participants, but optimal mental task combinations were intrinsically different among the participants. It would be ideal to use mental task combinations customized for each individual in developing a practical BCI system. However, the process to select the most suitable combinations of mental tasks is a time-consuming task, thereby making a BCI user feel exhausted even before using the BCI system. We think that our results can be utilized as a useful reference to simplify the procedure of selecting optimal mental task combinations, thereby reducing the time needed for preliminary tests. In our future studies, we will continue to test new mental tasks together with those that showed good classification performance in the present study.

Many previous NIRS studies have reported that brain activation generally induces an increase in [oxy-Hb] and a decrease in [deoxy-Hb], but in our results, inverted [oxy-Hb] and [deoxy-Hb] response patterns were sometimes observed in some mental task conditions (Figs. 8 and 9). Such inverted hemodynamic responses have also been frequently observed in other previous studies.^39,48–51 Particularly, two studies using the NIRS signals acquired during mental arithmetic tasks also showed a significant decrease in [oxy-Hb] and an increase in [deoxy-Hb] in the frontal lobe.^39,48

In this study, we used hemoglobin concentration values averaged over predefined time periods as features for classification because those features have been most widely used in NIRS-based BCI studies.^{17,18,24,27,43,44} Besides the mean values of hemoglobin concentrations, various types of features have been introduced in NIRS-based BCI studies, such as slope,^16,27,42 variance,^24,43 zero crossings,⁴³ and wavelet coefficients.²⁵ Apart from finding optimal mental task combinations, a systematic comparison of different feature types would be a meaningful research topic, which we would like to investigate in future studies.