Decoding semantic representations from functional near-infrared spectroscopy signals

Benjamin D. Zinszer; Laurie Bayet; Lauren L. Emberson; Rajeev D. S. Raizada; Richard N. Aslin

doi:10.1117/1.NPh.5.1.011003

23 August 2017 Decoding semantic representations from functional near-infrared spectroscopy signals

Benjamin D. Zinszer, Laurie Bayet, Lauren L. Emberson, Rajeev D. S. Raizada, Richard N. Aslin

Author Affiliations +

Neurophotonics, Vol. 5, Issue 1, 011003 (August 2017). https://doi.org/10.1117/1.NPh.5.1.011003

Abstract

This study uses representational similarity-based neural decoding to test whether semantic information elicited by words and pictures is encoded in functional near-infrared spectroscopy (fNIRS) data. In experiment 1, subjects passively viewed eight audiovisual word and picture stimuli for 15 min. Blood oxygen levels were measured using the Hitachi ETG-4000 fNIRS system with a posterior array over the occipital lobe and a left lateral array over the temporal lobe. Each participant’s response patterns were abstracted to representational similarity space and compared to the group average (excluding that subject, i.e., leave-one-out cross-validation) and to a distributional model of semantic representation. Mean accuracy for both decoding tasks significantly exceeded chance. In experiment 2, we compared three group-level models by averaging the similarity structures from sets of eight participants in each group. In these models, the posterior array was accurately decoded by the semantic model, while the lateral array was accurately decoded in the between-groups comparison. Our findings indicate that semantic representations are encoded in the fNIRS data, preserved across subjects, and decodable by an extrinsic representational model. These results are the first attempt to link the functional response pattern measured by fNIRS to higher-level representations of how words are related to each other.

1. Introduction

Multivariate statistical analyses for neuroimaging data have provided a new window into the contents of neurocognitive representations. Using multivariate pattern analysis (MVPA), cognitive neuroscientists can decode functional brain responses into signals of theoretical significance, such as representational models of word meaning or visual perception. This decoding is achieved by comparing the distributed neural response (that is, the multivariate pattern) to an analogous theoretical model. Such an approach was first used for decoding the meanings of words by Mitchell et al.,¹ who used a semantic model based on word co-occurrence frequencies to decode the meanings of concrete nouns from adult subjects using functional magnetic resonance imaging (fMRI). MVPA studies have demonstrated that neural activity encoding task-relevant information is both broadly distributed across the cortex and decodable into distinct, localizable components.²^–⁴ Thus, recent fMRI research has confirmed the utility of separable components of the neural response to classes of stimuli, contrasting representational models derived from computer vision, distributional (corpus-based) semantics, and conceptual features.¹^,⁵^–⁷

The adaptation of multivariate techniques, such as neural decoding, to other imaging modalities is critical for extending these advances in cognitive neuroscience beyond the MRI scanner and enabling research with participants who are ineligible or not well-suited for fMRI studies (e.g., infants, various clinical populations) as well as tasks that are simply not possible to investigate in the MR environment. In particular, infants usually cannot be scanned while awake, with very few exceptions,⁸^–¹⁰ limiting our ability to investigate the emergence of language in the brain during the most important developmental years. Brain imaging with functional near-infrared spectroscopy (fNIRS) is one especially promising approach for developmental neuroscience research. fNIRS has gained popularity for its advantages in testing infants and children, due to its tolerance for head and body motion and its applicability outside the confines of the MR environment. These unique advantages have also enabled research on social cooperation and face-to-face conversation.¹¹^–¹³

However, to date, the application of MVPA to fNIRS remains limited. One recent paper¹⁴ described the first demonstration of neural decoding in infants using fNIRS, based on representational similarity techniques developed for fMRI.¹⁵^,¹⁶ This work opens the door for using multivariate methods in developmental research, such as measuring semantic representations for words from the earliest stages of language acquisition to the adult-like state. Such fine-grained studies of language are now commonplace in fMRI but completely untested with fNIRS.

Although fNIRS measures a very similar physiological signal to fMRI, these techniques have important differences that affect the amount of information they provide for neural decoding. Most notably, the spatial resolution of fNIRS is considerably lower than fMRI, with individual channels covering only the surface of the cortex and sampling from regions a few centimeters in diameter. Further, fNIRS is sensitive to hemodynamic variations in the scalp and other tissues between the detector and the cortex. Some of this noncortical signal can be removed with statistical techniques (such as principal components analysis) to locate and subtract global responses, such as heartbeat or fluctuations in blood pressure. The relatively coarser spatial resolution (i.e., portion of the cortex covered by each measurement channel) and poorer signal-to-noise ratio of fNIRS compared to fMRI likely diminish the power of fNIRS for detecting the small changes in signal that would underlie comparisons among several semantic categories.

However, other aspects of fNIRS are very well-suited to multivariate pattern analyses. In particular, the fNIRS image is spatially specific. Although a single channel may cover a few square centimeters of the scalp, we can be highly confident that the hemodynamic response measured by that channel originates in that spatial area and is not susceptible to the volume conduction effects that apply to most electrophysiological measures, which have also been used in adult neural decoding.¹⁷ Although magnetoencephalography (MEG) has provided improvements in combined spatial and temporal resolution relative to electroencephalography and fMRI, respectively,¹⁸^,¹⁹ it remains very difficult to apply MEG to infants and young children, especially in visual or audiovisual paradigms. Further, fNIRS devices typically sample at a higher rate (10 Hz or more) than fMRI, providing many more observations of an individual event response curve than available from fMRI (i.e., in a 2-s time window, 20 fNIRS samples are obtained but typically only 1 fMRI sample). Thus, on the one hand, the spatial limitations of fNIRS may limit how much information is encoded in its relatively low-dimensional data (i.e., a few dozen channels versus thousands of voxels). On the other hand, fNIRS provides both a large amount of data about each channel and maintains spatial specificity.

In this study, we ask whether fNIRS data contain information suitable for neural decoding among subjects and for decoding based on extrinsic representational models. We present subjects with eight different matching audiovisual pairs (i.e., a spoken word and its corresponding picture) and attempt to identify specific stimulus representations based either on generalization from other subjects’ neural responses (between-subjects decoding) or on a distributional semantic model of the meanings of the stimulus words (semantic decoding). To achieve these goals, we integrate recently developed tools for representational similarity-based decoding in fMRI¹⁶^,²⁰ and similarity-based MVPA for fNIRS.¹⁴ We thus reveal the representational contents of the fNIRS signal encoded in multichannel response patterns, and we aim to motivate follow-up studies that can apply these methods to language research in infants and young children.

2. Experiment 1

2.1.

Method

2.1.1.

Participants

Eight adults (three males and five females) participated in a 15-min passive viewing and listening task. Participants were students and staff recruited from the Brain and Cognitive Sciences Department at the University of Rochester.

2.1.2.

Procedure

Informed consent procedures and experimental methods were approved by the institutional review board of the University of Rochester. Participants were presented with eight audiovisual stimuli, featuring a photograph of an object and simultaneous auditory presentation of the object’s name. Participants were asked to simply focus on the audiovisual stimuli and to think about the meaning of that stimulus or any memory it evoked. We directed the adult participants toward this more ecological activity (as compared to covert feature generation used in previous studies)¹ in order to best compare with children’s responses to the stimuli in a future experiment.

The stimuli were drawn from two broad categories (animals and body parts) and were all objects that would be familiar to infants, as one aim of this study is to validate the design for infants in future work. The objects were: bunny, bear, kitty, dog, mouth, foot, hand, and nose. Visual stimulus presentation lasted for 3 s, with the auditory word presented immediately at onset. A jittered 6- to 9-s interstimulus interval followed each trial.²¹ The interstimulus interval was composed of fireworks and a short musical clip, designed for infant paradigms and included for consistency between the adult data and future infant experiments. See Fig. 1 for illustration. This procedure was repeated over 12 blocks, with stimulus order randomized in each block. Participants’ blood oxygen levels were measured using the Hitachi ETG-4000 fNIRS system throughout the exposure.

2.1.3.

fNIRS measurement and preprocessing

The fNIRS probes were arranged in two arrays: 24 channels in a posterior $4 \times 4$ array approximately covering the occipital lobe and 18 channels in a lateral $3 \times 5$ array over the left temporal, parietal, and prefrontal lobes. (The lateral array typically provides 22 channels, but 4 of the anterior channels were excluded due to a broken laser.) Because the representational similarity-based analysis makes all of its channel-wise comparisons within-subject, only the overall positions of the arrays needed to be controlled across the subjects, allowing for variations in channel positions due to head size and shape. The lateral array was positioned directly above the left ear, approximately centered (anterior-to-posterior) over the ear and with the most anterior channel just beyond the hairline. The posterior array was centered on the back of the head, with the most inferior row of channels just over the inion.

Preprocessing of the fNIRS data was performed in Homer2.²² We computed a principal components analysis on the optical density data, removing the first component to reduce nonneural physiological signals. Next, the data were bandpass filtered (high pass: 0.01 Hz and low pass: 1.0 Hz). Oxygenated and deoxygenated hemoglobin concentrations were computed by Homer2 according to the modified Beer–Lambert law. The oxygenated hemoglobin concentration data were further processed using custom scripts to remove motion artifacts by masking a 1-s window around any observations that exceeded five standard deviations of the overall mean in that channel.

Finally, we performed a channel stability analysis adapted from fMRI decoding (voxel stability)¹ to determine which channels produced reliable responses across multiple blocks, independent of the discriminability among the stimulus classes. This procedure reduces the dimensionality of the fNIRS data and is intended to increase the signal-to-noise ratio by including channels that respond reliably while excluding channels that contain noise or change their response magnitude over the course of the experiment. For a given channel, the responses for each stimulus type are correlated across blocks. Thus, each block is represented by an $n$ -dimension vector for $n$ stimulus types, and the correlation between block 1 and block 2 is the Pearson $r$ statistic between these two $n$ -dimension vectors. These correlations are repeated for every possible pair of blocks, and the $r$ values are averaged to produce a mean stability value for that channel (further details are provided in the supplementary materials of Ref. 1).

Importantly, the channel stability procedure is independent of the differences between experimental conditions and thus does not need to be performed within each cross-validation loop, i.e., it does not lead to double dipping. For each subject, all channels are assigned stability scores, and the 50% most stable channels are retained (Fig. 2). The 50th percentile stability threshold was applied separately to each subject, so the specific set of channels is not constant across subjects. However, the representational similarity-based decoding procedures described below summarize fNIRS responses at the subject level. This abstraction away from channel-specific data distinguishes similarity-based decoding from other popular methods, and it allows comparisons among the subjects’ overall patterns of fNIRS response even when individual channels may differ in quality or location among individual subjects.

Fig. 1

Schematic of experimental procedures. (a) Eight visual stimuli depicting common objects were presented to participants for 3 s at a time, with immediate-onset auditory presentation of the object’s name. (b) Each block consisted of the eight randomly ordered stimuli, followed by a jittered 6- to 9-s interstimulus interval.

Fig. 2

Approximate positions of fNIRS probes on the left lateral and posterior areas of the head. Red circles indicate lasers and blue circles indicate detectors. Each channel (between a laser and detector) is marked by the number of participants (out of eight, total) for whom it was retained as one of the top 50% most stable channels, independent of decoding accuracy.

2.1.4.

Semantic model

We selected the compositional operations in semantic space model (COMPOSES), a well-known distributional semantic model,²³ to estimate semantic representations of the eight stimuli. The COMPOSES model describes the meanings of the stimulus words based on an analysis of the words’ use in large text corpora, primarily by measuring how often different words co-occur with each other. COMPOSES produces a 400-dimension vector representation of each word. While these representations are unique to the model, we applied representational similarity methods to abstract this model and the fNIRS data from their respective sources into a shared similarity space. That is, each of the eight COMPOSES vectors (one for each experimental stimulus) was correlated with the other seven COMPOSES vectors to yield an $8 \times 8$ similarity (cross correlation) matrix of Pearson’s correlation coefficients representing how similar the eight stimuli were to each other, according to that model [Fig. 3(c)]. An analogous $8 \times 8$ similarity matrix was derived for the fNIRS data (according to a method described below). This procedure allows the model and the fNIRS data to be directly compared in terms of their representational similarity structures [as captured by the $8 \times 8$ matrices, see Fig. 3(c)].

Fig. 3

(a) Between-subject decoding accuracy and (b) semantic model-based decoding accuracy for each of the eight participants. Chance level is 50%, and significance test is performed on the mean decoding accuracy across subjects. Semantic model-based decoding accuracy of the group-level model is also depicted. (c) Similarity matrices for the group-level fNIRS data and semantic model. (d) Classical multidimensional scaling of group-level fNIRS and semantic model (rotated, scaled, and overlaid).

2.1.5.

Analyses

Changes in oxygenated hemoglobin levels following each stimulus were epoched, baseline corrected, and averaged according to Emberson et al.’s¹⁴ MVPA procedures for fNIRS. Epoching was performed in a time window of 6.5 to 9.0 s after stimulus onset, following the optimal onset time identified for integrated audiovisual events,¹⁴ and terminating before the earliest onset of the next stimulus. Epoched and baseline corrected time series data for trials of each stimulus type were averaged across blocks to yield a mean response curve. The mean oxygenated hemoglobin level for this response curve was calculated in each of the 42 channels to produce a 42-dimension response pattern for each stimulus type. Each subject’s response patterns were abstracted to representational similarity space by cross correlating the eight 42-dimension vectors into an $8 \times 8$ similarity structure.

These semantic model- and fNIRS response-based representational similarity structures are compared at the subject level, decoding each subject’s fNIRS data using a group average similarity structure (excluding that subject, i.e., leave-one-out cross-validation) and using the semantic model. These comparisons are calculated based on the average accuracy of all possible two-alternative (pairwise) forced choice comparisons among the eight stimulus classes, as typically done in fMRI neural decoding studies.²⁰ Decoding accuracy provides a measure of the discriminability of the classes and the reliability of the similarity structures that support decoding across subjects or between the subjects and the model.

Figure 3(d) shows the strong similarities between a group-level fNIRS model (in blue) and the semantic model (in red) of the eight stimuli when these two structures are aligned in multidimensional space. The figure is generated using classical multidimensional scaling (MDS, MATLAB^® function cmdscale) to translate each $8 \times 8$ similarity matrix [from Fig. 3(c)] into distances between each stimulus in seven-dimensional space. After each of these models is independently scaled to best capture the relative similarities of the eight stimuli to one another (within the model), the two models’ structures can be compared for overall alignment. In this illustration, the semantic model is rotated through MDS space and scaled using Procrustes alignment to find the closest possible alignment with the neural data. The first two dimensions of these aligned structures are shown in Fig. 3(d). MATLAB^® scripts and sample data necessary to reproduce these analyses are provided on our Github page.²⁴

Importantly, this visualization illustrates the best possible alignment between the observed fNIRS and model-based structures when the correct matches for all eight classes are known and considered simultaneously. The pairwise decoding procedures rely only on the similarity data as shown in Fig. 3(c) and compare only two stimuli at a time without access to this structure-level alignment. Consequently, pairwise decoding is weaker than if this higher dimensional structural alignment was already known.

2.2.

Results

2.2.1.

Between-subjects fNIRS response-based decoding

Between-subjects decoding is performed with a leave-one-subject-out cross-validation procedure, wherein each subject is iteratively removed from the group and compared to the averaged group model. The group model is computed as the element-wise mean of the remaining subjects’ representational similarity structures (yielding an $8 \times 8$ group matrix). The average pairwise decoding accuracy in this between-subjects analysis was 0.70 [ $p = 0.01$ , Fig. 3(a)], compared to chance level of 0.50. All significance tests are based on the empirical null distribution for randomly permuted stimulus labels.¹⁴^,¹⁶^,²⁰ Null distributions for each significance test are provided with the demonstration code and can be regenerated using the code available on Github. $p$ -values are obtained by measuring the proportion of the null distribution that is greater than or equal to the observed accuracy.

Individual level performance was variable, ranging from 0.39 to 0.96. This variability suggests that either the individual subjects’ data quality was inconsistent or some subjects’ response patterns differed from the group. In the next analysis, we ask whether the COMPOSES semantic model explains the subjects’ fNIRS response patterns.

2.2.2.

Semantic model-based decoding

The COMPOSES semantic model decoded the fNIRS data from the stable channels with a mean accuracy of 0.66 ( $p = 0.03$ ) across subjects. Figure 3(b) shows individual subjects’ decoding accuracy based on the COMPOSES model. We also investigated whether this result could be improved by averaging across the eight subjects’ similarity structures to test the group model’s semantic decoding. The semantic model decoded the group model data with an accuracy of 0.75, a considerable gain over the mean individual level performance, although this value did not achieve conventional significance when tested as a single-point observation. Instead, we return to this group model in the next experiment.

Finally, we performed some additional analyses that explore these results in greater depth, beyond the scope of this paper. We contrast the roles of within-category (e.g., foot versus nose) and between-category (e.g., foot versus kitty) comparisons to decoding accuracy, and we compare the performance of a second distributional semantic model to the COMPOSES model reported here. These supplemental materials are available for download on our Github page.

2.3.

Discussion

Experiment 1 illustrates neural decoding of fNIRS data using a representational similarity-based method. The similarity structures used by this method allowed us to compare across different subjects’ fNIRS data, to compare individual subjects’ fNIRS data to a semantic model, and to combine subjects’ patterns into a group model, despite individual variation in channel location and data quality. All of these comparisons are made possible by the fact that similarity-based multivariate pattern analyses abstract the individual’s neural data out of their unique anatomical space and into similarity space.

The high between-subjects decoding accuracy illustrates the effectiveness of using these similarity structures to overcome the limitations of fNIRS imaging and to find reliable stimulus-dependent response patterns that are preserved among individual subjects. This finding indicates that a shared set of representations for the eight stimuli is encoded in the fNIRS data for these subjects, and the across-subject reliability is sufficient to predict the stimulus labels of a given subject based on a group model of the other seven subjects.

A significant portion of these fNIRS patterns was also explained by specific properties of the stimulus, as indicated by the success of the semantic model at decoding both individual subjects’ neural response patterns and decoding the group-level model. In the next experiment, we ask whether this group-level fNIRS response pattern can be replicated with additional groups of participants, whether these groups can also be explained by the semantic model, and whether the two fNIRS arrays may carry different types of information.

3. Experiment 2

In the foregoing experiment, we demonstrated that fNIRS data contained neural response patterns that were (1) preserved across subjects, (2) decodable by a semantic model, and (3) can be meaningfully combined across subjects to improve signal-to-noise ratio in a group-level model. In this experiment, we build upon these findings to further test the reliability of the group-level fNIRS response patterns by using between-groups decoding (analogous to between-subjects decoding) and by applying the semantic model to these group models.

Further, we explore the possibility that different types of information are encoded in different fNIRS channels. Although exact channel placement varies across participants, the similarity-based metrics allow us to compare on a larger scale: array-level response patterns should be strongly preserved among subjects because the arrays were consistently placed over comparable anatomical regions and contain several channels each from which to perform decoding analyses.