2.1.1.

Contrast

The primary assessment of the effect of a given absorption change in the compartment $i$ on the measurand $M$ is to consider the difference with respect to the baseline state (prior to the absorption change) $M_0$

Which of these options is more appropriate depends on the particular measurand $M$. For example, for the photon count $N_k$ in a $k$’th time window, only the ratio $\mathrm{\Delta }{N}_{k,i}/N_{k,0}$ is relevant, whereas the attenuation related to a time window requires that the absolute difference $\mathrm{\Delta }{A}_{k,i}=-\ln (N_{k,i}/N_{k,0})$ be considered. In other cases, e.g., for moments of DTOFs, both options can be applied.

A valid comparison of contrast is feasible only within measurands of the same kind, for example, within various ratios of photon counts in time windows. Comparison of measurands of different kinds is possible on the basis of the following tests: contrast-to-noise ratio, depth selectivity, lateral spatial resolution, and linearity.

2.1.2.

Contrast-to-noise ratio

The detectability of a small absorption change $\mathrm{\Delta }{\mu }_{a,i}$ in the compartment $i$ depends on the contrast-to-noise ratio ${\mathrm{CNR}}_i$, i.e., the ratio between the corresponding change in the measurand $\mathrm{\Delta }{M}_i$ and the uncertainty of $M$ due to random effects.

2.2.1.

Depth selectivity

This test addresses the capability of instruments and/or methods of data analysis to distinguish between absorption changes occurring in different compartments of the head, in particular, in the cortex versus the overlying tissues. It compares the sensitivity of the measurand $M$ to a small absorption change $\mathrm{\Delta }{\mu }_a$ in the lower compartment (index 2, corresponding to the cortex) with the sensitivity to an absorption change in the upper compartment (index 1, corresponding to extracerebral, superficial tissue) by considering the ratio

For ease of implementation, the reduced scattering coefficient ${\mu }_s^{\prime }$ is kept constant and equal in both compartments. The sensitivity ratio $S_{\mathrm{2,1}}$ is derived from three measurements, $M({\mu }_{a,1},{\mu }_{a,2})=M_0$ (baseline state), $M({\mu }_{a,1},{\mu }_{a,2}+\mathrm{\Delta }{\mu }_{a,2})$ with a change in the lower compartment only, and $M({\mu }_{a,1}+\mathrm{\Delta }{\mu }_{a,1},{\mu }_{a,2})$ with a change in the upper compartment only. For $\mathrm{\Delta }{\mu }_{a,1}=\mathrm{\Delta }{\mu }_{a,2}$, the ratio $S_{\mathrm{2,1}}$ equals the ratio of the corresponding contrasts, which is valid for both cases of contrast definitions, see Eqs. (1) and (2).

The ratio $S_{\mathrm{2,1}}$ is dimensionless and can be used to compare measurands of any kind and dimension. Ideally, a measurand that is sensitive to absorption changes in the cortex should only exhibit an infinitely large value of $S_{\mathrm{2,1}}$. The magnitude of $S_{\mathrm{2,1}}$ critically depends on the thickness of the upper compartment.

2.2.2.

Lateral spatial resolution

Lateral spatial resolution is determined as a spatial point spread function (PSF) by measuring the diffuse reflectance for a small absorber moved across this region. It is quantified by the full width at half maximum (FWHM) of the PSF related to a measurand $M$ along two perpendicular directions parallel to the surface.

2.3.1.

Accuracy

The accuracy of a measured absorption change $\mathrm{\Delta }{\tilde{\mu }}_{a,i}$ is characterized as a relative measurement error, i.e., the relative deviation from its (conventional) true value $\mathrm{\Delta }{\mu }_{a,i}$.

The index $i$ denotes the compartment in which a known absorption change $\mathrm{\Delta }{\mu }_{a,i}$ was realized. The value of $\mathrm{\Delta }{\tilde{\mu }}_{a,i}$ is available only if an absorption change can be derived by the application of a reconstruction algorithm to the measured DTOFs.

The accuracy depends on the measurement as well as on the reconstruction algorithm. Practically, the conventional true value $\mathrm{\Delta }{\mu }_{a,i}$ realized in an inhomogeneous phantom can be obtained by prior accurate characterization of the individual turbid materials in homogeneous phantoms. The absolute (baseline) optical properties of the materials used should also be known and kept fixed when comparing the accuracy of $\mathrm{\Delta }{\mu }_a$ measurements performed with different instruments and/or reconstruction algorithms.

2.3.2.

Linearity

To assess the linearity of the change in the measurand $M$ with respect to an underlying absorption change $\mathrm{\Delta }{\mu }_{a,i}$ in the compartment $i$, the values for $\mathrm{\Delta }M$ (together with their uncertainty) are plotted as a function of $\mathrm{\Delta }{\mu }_{a,i}$. The linearity range is defined as the maximum value of $\mathrm{\Delta }{\mu }_{a,i}$ for which the deviation from proportionality does not exceed a certain percentage of the respective $\mathrm{\Delta }M$. This test requires several absorption changes to be realized, starting from small values and a sufficiently low uncertainty of $\mathrm{\Delta }M$ due to random effects.

3.1.1.

Liquid phantoms

The decision to use well-characterized liquid phantoms for the performance characterization of instruments in the present study was based on the following considerations:

A detailed procedure was developed at the University of Florence (UNIFI) to prepare liquid phantoms with known optical properties based on water dilutions of Intralipid as a diffusively scattering component and India ink as an absorber. The amounts of components to be mixed were solely determined by accurate weighing, which ensured that the concentrations were known with high accuracy. UNIFI provided the other partners with Intralipid and India ink from the same batches and with optical properties characterized with an uncertainty of $<3\%$. The mixtures had to be freshly prepared on the day of the experiment. Accurate weighing and homogeneous mixing of the components is mandatory. It should be noted that these phantom materials were also characterized in an interlaboratory comparison of nine laboratories of the nEUROPt consortium and beyond. The intrinsic absorption coefficient of India ink and the intrinsic reduced scattering coefficient of Intralipid-20% were determined with an uncertainty of $\sim 2\%$ or better.²¹

Modular scattering cells made of black polyvinyl chloride (PVC) with small plexiglass windows were provided by UNIFI for measurements of time-resolved reflectance in homogeneous and layered geometry (see Refs. 18 and 22). For the implementation of the nEUROPt protocol, black front walls of 2 mm thickness were used, with three transparent windows (diameter 7 mm, adapted to the diameter of the fiber bundles used) for a source optode and a detector optode positioned 20 and 30 mm apart. For the cell, spacers of 10 and 30 mm thickness were provided. The effects of the finite lateral dimensions of the cell, the shape, and the refractive index mismatch of the inclusion were studied in detail by Monte Carlo (MC) simulations.²² The inner dimensions of the cell were 120 mm width, 145 mm height, and 70 mm thickness (with three spacers). A Mylar foil of 30 μm thickness could optionally be inserted as a separator with minimal perturbation of light propagation to realize the two-layer geometry.¹⁸ During data analysis of the measurement campaign, it turned out that it is not easy to maintain a constant and reproducible thickness of the upper layer (see Sec. 5.2). Therefore, a dedicated mounting plate for the Mylar foil was designed and manufactured by Physikalisch-Technische Bundesanstalt (PTB) for future use with a smaller area ($60\mathrm{mm}\times 80\mathrm{mm}$) to be covered by the foil. This arrangement was used when characterizing the instrument POLIMI_2.

3.1.2.

Absorbing objects

To mimic the perturbation due to small localized variations of the absorption coefficient for the nEUROPt protocol, UNIFI proposed the use of small black PVC cylinders immersed in the homogeneous liquid phantom.²² They can, by far, more easily be manufactured and reproduced than small inhomogeneities made of a scattering and moderately absorbing material with well-known optical properties. The major idea behind this approach is the equivalence of the perturbation by a small black inclusion and by a certain moderate absorption change in a given volume. For details, see Martelli et al.¹⁷ Thus, the small black cylinders can be used as a kind of universal inclusions provided their position is sufficiently far ($>10\mathrm{mm}$) from the source and detector and from the boundary of the medium. PVC cylinders of various sizes were provided by UNIFI, with dimensions (diameter equal to height) of 3.2, 4, 5, 6.8, and 8.6 mm, corresponding to volumes $V_{\text{incl}}/{\mathrm{mm}}^3=25$, 50, 100, 250, and 500. They were held by thin, rigid metallic wires (0.5 mm music wire) that were painted white in order to reduce their influence on the measurement. The perturbation by the black cylinders can be directly matched to the equivalent finite $\mathrm{\Delta }{\mu }_a$ changes over a certain larger volume (for instance, for a reference volume $V_0=1000{\mathrm{mm}}^3$, the equivalent values for the five cylinders were found to be $\mathrm{\Delta }{\mu }_a/{\mathrm{mm}}^{-1}=0.0056$, 0.0087, 0.015, 0.037, and 0.094, within the restrictions regarding the position as mentioned above. Thus, linearity plots can be drawn against the equivalent $\mathrm{\Delta }{\mu }_a$.

5.1.1.

Depth-dependent contrast

Contrast and noise measured as a function of the depth of a black inclusion in the liquid phantom are presented in Fig. 2 for two instrumental configurations, for analysis of time windows. Figure 2(a) clearly illustrates the advantages of the time-resolved measurement to detect absorption changes in the brain. Photons with a short time of flight are only sensitive to shallow absorption changes. For late time windows, the highest contrast is found deeply below the surface, in this example at $\sim 15\mathrm{mm}$. The error bars represent noise as obtained from the standard deviation of 100 repetitions of the measurement at a 1-s duration and a count rate of $\sim 1·{10}^6{\mathrm{s}}^{-1}$. Only for the latest time window, where the photon count is low, do they exceed the size of the markers. The comparison of Figs. 2(a) and 2(b) highlights the influence of the IRF on the contrast, in particular, for late photons (see also related simulations in Ref. 10). The depth-dependent contrast for instrument PTB_2 for late times (eighth time window, ranging from 3500 to 4000 ps) is similar to that for the third time window. The reason is the existence of a substantial late afterpeak in the IRF of PTB_2 (see Fig. 1). Meanwhile, the hybrid detector employed in the setup PTB_4 together with a supercontinuum laser has a good time resolution and an IRF without afterpeaks. The contrast for a deep inclusion, e.g., at $z=20\mathrm{mm}$, is considerably larger for PTB_4 compared to PTB_2.

Fig. 2

Contrast of photon counts in time windows measured as a function of depth position $z$ of a black polyvinyl chloride (PVC) cylinder of $100{\mathrm{mm}}^3$ volume, for instrumental configurations PTB_4 (a) and PTB_2 (b). The numbers indicate consecutive time windows of 500 ps width; error bars represent noise as obtained from the standard deviation of repeated measurements.

Figure 3 provides another representation of the results of the same measurement. For the inclusion positioned at shallow depths, the contrast decreases with increasing time [Fig. 3(a)], whereas a deep inclusion becomes detectable only at later times [Fig. 3(b)]. For the configurations PTB_1 and PTB_2, a deviation from this behavior is observed at late times, i.e., too high a contrast for the shallow inclusion and too low a contrast for the deep inclusion. This finding can be explained by the fact that an afterpeak in the IRF (see Fig. 1) effectively transfers early photons to late times. At the same time, these falsely assigned photons lead to a decreased noise and deviations in the contrast-to-noise ratio [Fig. 3(c)]. A comparison of Figs. 3(c) and 3(d) reveals a considerably lower maximum CNR for the deep versus shallow position of the inclusion. For the inclusion at $z=6\mathrm{mm}$, the maximum contrast and CNR are both found at early times. The maximum CNR for $z=16\mathrm{mm}$, however, is found at intermediate times. At late times, where the contrast is highest, the low photon count at late times causes the CNR to drop. The results presented in Figs. 2 and 3 illustrate that contrast and CNR alone are not sufficient to characterize the discrimination between a superficial and a deep absorption change. For this characteristic, we refer to depth selectivity, see Sec. 5.2.

Fig. 3

Contrast [(a) and (b)] and contrast-to-noise ratio [(c) and (d)] for the black cylinder of $100{\mathrm{mm}}^3$ volume as a function of time, for depths $z=6\mathrm{mm}$ [(a) and (c)] and $z=16\mathrm{mm}$ [(b) and (d)]. The results are plotted for the configurations PTB_1, PTB_2, and PTB_4.

As an example of analysis of moments, the depth-dependent contrast for variance derived from DTOFs is shown in Fig. 4 for three of the black cylinders. Unlike in the case of a time-window analysis, the results for the three different experimental configurations applied in the same measurement almost exactly match. This finding is explained by the fact that measurands based on differences in variance (as well as mean time of flight) are virtually independent of the IRF.³⁵ In addition, the measured variance contrast agrees very well with the results of MC simulations performed according to Ref. 33 for black spheres of equal volume (data not shown, see Ref. 22). Note the changing sign of variance contrast between 10- and 12-mm depths, a behavior that is typical for normalized moments.

Fig. 4

Difference in variance for $z$ scans of black PVC cylinders of various sizes (squares: $25{\mathrm{mm}}^3$, triangles: $100{\mathrm{mm}}^3$, circles: $250{\mathrm{mm}}^3$). Line color indicates different experimental configurations.

5.2.1.

Contrast

The measurements on the two-layered phantom allow the contrast of laterally extended absorption changes in a superficial and a deep compartment to be compared. Results of a time-window analysis of the measurements on the two-layer phantom with an upper layer of nominal thickness of 10 mm are presented in Fig. 5, together with the results of two-layer simulations based on Ref. 34. The absorption is changed in each of the compartments separately; the corresponding results are plotted in the upper and lower rows. Note that the range of the absorption change is rather large, ${\mu }_a$ is changed from $0.01{\mathrm{mm}}^{-1}$ (baseline) to $0.02{\mathrm{mm}}^{-1}$. It is not surprising that the contrast is not linear in $\mathrm{\Delta }{\mu }_a={\mu }_a-0.01{\mathrm{mm}}^{-1}$ in the whole range.

Fig. 5

Contrasts of selected time windows of 500 ps width (number given in the legend) as a function of a change in ${\mu }_{a,1}$ in the upper layer (upper row) and ${\mu }_{a,2}$ in the lower layer (lower row) of the two-layered phantom with nominal thickness of the upper layer $d=10\mathrm{mm}$. First column: two-layer simulation based on Ref. 34, second column: PTB_1, third column: PTB_2, fourth column: POLIMI_2.

The contrast for changes in the upper layer is comparable with the simulated data for all instruments, whereas the experimental results for changes in the lower layer show less contrast than expected from the simulations. A major reason is the finite width of the IRF that has not been accounted for in the simulations. In general, the sensitivity to absorption changes in the lower layer is higher for later photons, as observed in the case of POLIMI_2. The experimental data for the instruments PTB_1 and PTB_2 show, however, that the maximum contrast for the lower layer is achieved for intermediate time windows. For the latest time windows, the contrast decreases again. This behavior can be explained by the influence of afterpeaks in the IRF (see Fig. 1 and Ref. 10).

The contrasts for moments obtained on the two-layered phantom with the instrument IBIB_1 is depicted in Fig. 6, together with simulations with the same nominal optical properties and thickness of the upper layer. Measured and simulated contrasts agree very well despite the fact that the IRF has not been taken into account in the simulations. As already discussed in Sec. 5.1, this behavior is expected since differences of moments are virtually independent of the IRF.³⁵

Fig. 6

Contrasts of moments, (a) integral, (b) mean time of flight, (c) variance, as a function of a change in ${\mu }_{a,1}$ in the upper layer (up triangles, gray) and ${\mu }_{a,2}$ in the lower layer (down triangles, black) for the instrument IBIB_1. Both integration limits for moments were set at 1% of the maximum. Dotted lines: simulations based on Ref. 34.

In general, the contrast obtained from $N_{\mathrm{tot}}$ is larger in the upper layer (by about a factor of 3), and the contrast based on $m_1$ is approximately the same in both layers, whereas the contrast for $V$ is larger (by about a factor of 2) for absorption changes in the lower layer. A quantitative comparison of these results is performed in the following section.

5.2.2.

Depth selectivity

The definition of depth selectivity [Eq. (4)] based on a ratio is a way to compare different measurands even of different dimensions (see the different moments in Fig. 6) with respect to the discrimination between deep and superficial changes. This ratio is formed from data such as those displayed in Figs. 5 and 6 for the lower and upper compartments. In Fig. 7, the same quantity $S_{\mathrm{2,1}}$ is plotted for analysis by time windows [Fig. 7(a)] and moments [Fig. 7(b)]. $S_{\mathrm{2,1}}$ was obtained as the ratio of the slopes of the contrast curves for small absorption changes, i.e., for ${\mu }_a$ values in the interval [$0.010{\mathrm{mm}}^{-1}$, $0.012{\mathrm{mm}}^{-1}$]. The consolidated plots in Fig. 7 include the results obtained for several different instruments and configurations together with results obtained from simulations. Comparing the plots for the instrumental configurations PTB_1, PTB_2, and PTB_3, which differ in the type of detector, reveals the different influence of the IRF with both approaches of analysis. For time windows [Fig. 7(a)], the maximum achievable depth selectivity for late photons is substantially degraded if the IRF exhibits remarkable afterpeaks as in case of the instruments PTB_1 and PTB_2. The existence of afterpeaks causes a fraction of early photons to be detected at late times. Such afterpeaks are not present in the case of the hybrid detectors in PTB_3 and POLIMI_2. These instruments provided the best depth selectivity at late times. Moreover, in these cases, the courses as a function of time essentially follow that for the simulated data where no IRF was taken into account.

Fig. 7

Depth selectivity, i.e., the ratio of contrasts for small absorption changes in the lower layer and in the upper layer, for time windows (a) and moments (b). The positions of the symbols in (a) correspond to the center of the respective time windows.

For moments [Fig. 7(b)], the influence of the IRF is expected to vanish, as discussed above in Sec. 5.1. This can actually be observed when comparing the results for the three PTB detectors that were obtained together in the same phantom experiment. It should be noted that here the integration limits were set at 0.1% of the maximum (otherwise 1%) to avoid cutting within the region influenced by the afterpeak for PTB_2. Discrepancies between the measurements by different groups are most likely due to inaccuracies in the thickness of the upper layer caused by difficulty in flattening the inelastic Mylar foil that separates the layers. The comparison with corresponding two-layer simulations suggests that in the case of PTB, the thickness of the upper layer was most likely $\sim 9\mathrm{mm}$ instead of 10 mm. A decreased thickness of the upper layer mainly leads to an increase in contrast for changes in the lower layer and, thus, to an overestimation of depth selectivity. The inaccuracy of the thickness hampers the comparison between different experiments. An improved mounting of the Mylar foil employing the mounting plate mentioned in Sec. 3.1 could solve this problem. This mounting plate was applied in the later experiment with the instrument POLIMI_2, which yielded depth selectivities for moments rather close to the simulated values.

Overall, the highest depth selectivity is obtained for variance. It should be noted that the depth selectivity of the time-window approach can be substantially improved by considering ratios of photon counts in late to early time windows.