1.

Introduction

Fluorescence lifetime imaging microscopy (FLIM) is an intrinsically quantitative tool to image the lifetime of molecular fluorescence. Changes in fluorescence lifetime are an important biomedical indicator, as the fluorescence lifetime can change, for instance, in the presence of oxygen or ions,^1,2 changes in local pH,³ and interactions between proteins in living cells.^4,5

There are two main approaches to estimate fluorescence lifetime, one in the time domain (TD), and the other in the frequency domain (FD).⁶ In TD-FLIM, a train of pulsed light, the width of which should be significantly smaller than the decay time of the fluorescent sample, is used for excitation. The decay curve of the emission photons is detected using a time-resolved detection system.⁷8.^–9 It is an inherently direct measurement of the fluorescence decay. The data analysis of TD-FLIM is typically achieved by fitting the experimental data to a linear combination of decaying exponentials, as shown in Eq. (1):

The values of ${\tau }_k$ represent the different lifetime components in the sample under study, and the values of $p_k$ are their proportional contributions. The fitting process not only costs computation time but generally requires a high level of expertise to obtain reliable results.¹⁰ The TD-FLIM system is also relatively expensive, since it requires short pulsed lasers and fast, sensitive detection systems.

An alternative way is through the frequency-domain approach. FD-FLIM uses periodically modulated light for the excitation and deduces the lifetime values from the phase change and/or the modulation depth change between excitation and emission signals, as shown in Eqs. (2) and (3):⁶

To retrieve the phase change and the modulation depth change, the sensitivity of the detector is modulated at the same frequency, and a series of images are taken at different phase offsets.¹⁸19.^–20 The current state-of-the-art FD-FLIM system requires an image intensifier, the use of which is necessary for low light levels and MHz demodulation frequencies.^11,17,21 The demodulation is done by controlling the voltage of the photo cathode in the image intensifier. Although this technique is well developed and has been commercialized, there are still several fundamental drawbacks introduced by this technique. These will be described in the following section.

We propose improving FD-FLIM instrumentation by replacing the image intensifier–based charge coupled device (CCD) camera with an application-specific CCD design. We have designed, built, and tested such a CCD image camera that can be modulated at the pixel level, which we have named the MEM-FLIM camera (modulated electron multiplied all-solid-camera for fluorescence lifetime imaging microscopy). In the current version of our CCD design, the electron multiplication principle has not yet been implemented. This will occur in the next version.

2.

Theory and Principle

In conventional FD-FLIM systems, the fluorescent molecule is illuminated by an amplitude-modulated light source, and the generated fluorescent light is demodulated by using a microchannel plate (MCP) image intensifier. The main disadvantage of FD-FLIM is the requirement of the MCP image intensifier. The image intensifier consists of a photo cathode that converts the incident photons to electrons, an MCP that accelerates and multiplies the electrons, and a phosphorus screen that converts the electrons back to photons. An illustration of the image intensifier structure is shown in Fig. 1. The image intensifier is then coupled to a CCD image sensor by using a fiber optic taper. The demodulation is done by changing the photo cathode voltage.

Fig. 1

Illustration of the image intensifier.

As we see in Fig. 1, high voltages up to several kilovolts are needed to operate the image intensifier. The spatial resolution is compromised by the photo cathode and the MCP.²² To modulate the sensitivity of the camera, a periodical demodulation signal ($V_{pc}$) is applied to the cathode. A higher voltage on the photo cathode compared to the one on the entrance of MCP will let none of the electrons through, whereas a lower voltage will open the intensifier. This means that during the demodulation, half of the signal is lost. High voltage, up to several kilovolts, will be applied on the phosphorus screen ($V_{ps}$). Furthermore, the system is relatively costly, bulky, and vulnerable to overexposure.

Our noncooled MEM-FLIM sensor has been designed for pixel-level modulation, which means that the demodulation is done on the camera pixel itself, instead of on the image intensifier, which sits in front of the CCD camera in the conventional method. The principle of the MEM-FLIM camera at the pixel level is illustrated in Fig. 2. Demodulation signals, which have a 180-deg phase difference, are applied on two adjacent toggling gates of one pixel. In the first half of the demodulation cycle, the photo-generated charge will be transferred to one storage gate (STG), and in the second half of the cycle to the other STG. In this way, two phase images are obtained in one integration and read-out cycle. So the readout image contains these two phase images interleaved with each other, which we called “phase one” image and “phase two” image, as shown in Fig. 2(c). There is no dedicated register for transferring the charge to the horizontal register during the readout. The photo gates (PGs), toggle gates (TGs), STGs, and barrier gates (BGs) are all used for vertical transport during the readout. The chip, therefore, resembles a frame transfer sensor.

Fig. 2

Principle of modulated, electron multiplied fluorescence lifetime imaging microscope (MEM-FLIM) camera. (a), Toggling principle at pixel level. (b), Architecture of the chip level. BG: blocking gate; STG: storage gate; TG: toggling gate; and PG: photo gate. (c), Illustration of two phase images interleaved with each other.

The incoming light is thereby captured by modulated pixels, recording two phase images at once. This is in contrast to an image intensifier with a duty cycle of about 50% when recording a single phase image. By removing the intensifier and fiber/lens coupling from the camera, a noise source is eliminated as well as a source of image distortion.

Our system has been designed with a variable integration time $T_0$ such that 1 ms $\le T_0$. The choice of $T_0$ is related to the strength of the fluorescent image. The image is then read out before the next integration cycle begins. The time for integration plus readout time $T_R$ plus a user-chosen delay $T_{\mathrm{DL}}$ is referred to as the frame time $T_1$, that is

We are not the first group to use the approach of demodulation at the pixel level. In 2002, Mitchell et al.^23,24 demonstrated the feasibility of measuring fluorescence lifetime with a modified CCD camera. By modulating the gain of a CCD at a frequency of 100 to 500 KHz, images were recorded with an increasing delay. This camera, however, was not really suitable for FLIM since the maximum modulation frequency could only be 500 kHz. The “sweet spot” for frequency in an FD-FLIM system is approximately $f_o=1/(2\pi \tau )$, which for $\tau =5\mathrm{ns}$ translates to about 30 MHz. The value of 500 kHz is clearly too low.

In 2003, Nishikata et al.²⁵ succeeded in taking two phase images simultaneously at a modulation frequency of 16 kHz. Again, the modulation frequency is much too low but the two-phase approach can be found in our work as well.

Later Esposito et al.^26,27 developed this technique further and performed FLIM measurements at 20 MHz using a CCD/complementary oxide semiconductor (CMOS) hybrid sensor (SwissRanger SR-2). The SR-2 was originally developed for full-field 3-D vision in real time.²⁸ Later in this manuscript, we will compare the performance of this camera to our implementation for FD-FLIM.

A solid-state camera can also be used in TD-FLIM. The MEGA frame project started in 2006, and is time domain based. A CMOS single-photon avalanche diode (SPAD)-based camera has been developed for TD-FLIM.^29,30 The prototype camera has $128\times 128\text{pixels}$.

3.

Materials and Methods

3.2.

Camera Characteristics: Background

3.2.1.

Linearity of photometric response

It is extremely convenient for a scientific camera to have a linear response to the incident light. The linearity of photometric response of a camera is gauged by the coefficient of regression, calculated from a straight-line fit of intensity readout data under various exposure times. Below saturation, the CCD is usually photometrically linear. The closer the coefficient of regression is to 1, the better the linearity of the camera.

3.2.2.

Sampling density

Sampling density refers to the physical scale between pixels in the digitized microscope image. An image with $a\times a\text{pixels}$ that covers a physical area of $L\times L{\mu \mathrm{m}}^2$ has a sampling density of $a/L$ samples per micron in both directions. Equivalently, the sample distance along any of these directions is $L/a\mu \mathrm{m}$. The sampling densities along both the horizontal and the vertical directions are preferably the same.³³ The sampling densities of the MEM-FLIM camera and the reference camera are measured by using a stage micrometer. The $20\times$, 0.5-NA objective lens is used in the experiment.

3.2.3.

Resolution

Owing to inevitable aberrations and diffraction phenomena, the image of an object observed with an optical system will be somewhat degraded. As a rule, the bright areas in the image will not be as bright as in the original pattern, and dark areas will not be as dark as in the original pattern. There will be a smooth transition along the originally high-contrast edges. The optical transfer function (OTF) is a commonly used quantity for describing the resolution and performance of an optical system.³⁴ One way to measure the OTF is to use a test pattern such as that shown in Fig. 3. A higher OTF indicates a better performance of an optical system. Using the method described in Ref. 33, the OTF can be calculated from the edge response. Our measurements are made in both the horizontal direction and the vertical direction. The MEM-FLIM and reference FLIM systems share the same system settings (microscope, filter cube, illumination) except that the fluorescence emission is switched and directed into the two different camera ports. Thus the OTF directly reflects the performance of the camera. All OTF measurements have been made with a magnification of $100\times$, 0.6 NA objective lens and a 180 ms integration time. The test pattern was illuminated via transmitted white light.

Fig. 3

Test pattern used for measuring the optical transfer function (OTF) and the intensity along the test bars.

The OTF can be influenced by effects such as the misdiffusion of the electrons generated outside the depletion layer, nonideal charge transfer effects, the photosensitivity of the device, and so on.³⁵

3.2.4.

Noise

The main possible noise sources for digitized fluorescence images can be characterized as: photon noise due to the random arrival of photons, dark current noise due to random generation of electrons by thermal vibrations, readout noise due to the on-chip amplifier that converts the electrons into a change in analog voltage, and quantization noise due to quantizing the pixels of a sensed image into a number of discrete levels.

Photon noise

The fundamental quantum physics of photon production determines that the photon noise $N_p$ is Poisson distributed,³⁶ as shown in Eq. (4):

(4)

$$p(n|{\mu }_p)=\frac{{\mu }_p^n{e}^{-{\mu }_p}}{n!}n=0,1,2,3,\dots ,$$

where ${\mu }_p$ is the expected number of photons during a given interval and $n$ is the number of random occurrences. To validate the Poisson distribution assumption, we make use of an important characteristic of Poisson distribution: $\langle N_p\rangle ={\mu }_p={\sigma }_p^2$.

The Poisson distribution assumption of the photon noise will be validated using the following method. Two images are taken under the same illumination condition. The photon noise level is determined by subtracting these two images so that deterministic pixel variability in the image (e.g., shading) can be eliminated. Half of the variance in the difference image is used to represent the variance of a single pixel. To confirm the assumption that the noise source of the camera is Poisson distributed, we take two images and obtain the difference image under different illumination intensities. If the assumption is valid, then half of the variance in the difference image reflects the noise level in the single image. The variance for a Poisson distribution should be linear with the mean intensity.⁶

Dark current noise

Dark current noise $N_d$ refers to the creation of electron-hole pairs due to thermal vibrations.³⁷ It is intrinsic to semiconductors and is a stochastic process with a Poisson distribution and thus $\langle N_D\rangle ={\mu }_d={\sigma }_d^2$. It reduces the dynamic range of the camera, since it produces an offset to the readout value, and can be a substantial source of noise. Cooling the camera reduces the dark current significantly.

The dark current can be influenced by the previously defined integration time ($T_0$) and frame time ($T_1$) in the MEM-FLIM camera, and it is, therefore, necessary to evaluate their individual contributions. This can be accomplished by varying the aforementioned $T_{\mathrm{DL}}$. The linearity of the dark current noise in the integration time is also validated using the same method as in Sec. 3.2.1. Because the name “dark current” refers to the electron-hole pairs that are created when the camera is not exposed to light, measuring dark current is relatively simple and requires no optical setup.

Readout noise

Readout noise $N_r$ is a fundamental trait of CCD cameras caused by the CCD on-chip electronics before digitizing. It is independent of integration time but dependent on the readout bandwidth. By validating the linearity of the dark current noise to the integration time, the readout noise with a mean of $\langle {\mu }_r\rangle =0$ and a variance ${\sigma }_r^2$ can be deduced from the fitting by extrapolating the noise level in the limit as the integration time goes to zero.

Quantization noise

Quantization noise $N_q$ is the roundoff error when the analog-to-digital converter (ADC) converts a sensed image to a finite number of discrete levels, and thus $\langle N_q\rangle =0$ and $\langle N_q^2\rangle ={\sigma }_q^2$. Quantization noise is inherent in the quantization process. For a well-designed ADC with the number of bits $b$ higher than 8 (the MEM-FLIM camera has 14 bits, and the reference camera has 12 bits), the quantization noise can be ignored, as the signal-to-noise ratio (SNR) is given by $6b+11\mathrm{dB}$.^6,37,38

3.2.5.

Sensitivity

Sensitivity relates the A/D converter units (ADU) of a digital camera system to the number of photo-electrons produced by incident photons reaching the pixels.

Sensitivity

Sensitivity measures a camera’s ability to convert photo-electrons to ADUs. For a photon-limited signal, the conversion factor $G$ from photo-electrons to ADUs can be determined by Eq. (5):³⁷

(5)

$$G=\left[\frac{\mathrm{var}(I_1-I_2)}{2}\right]/\overline{I},$$

where $I_1$ and $I_2$ are two images taken under the same illumination condition. $\overline{I}$ is the mean intensity over a uniformly illuminated field. $G$, in the unit of $\mathrm{ADU}/{\mathrm{e}}^{-}$, is indicated by the slope of the fitted linear curve to the noise level for different intensities.

Comparing camera sensitivities

To compare the sensitivities of the two cameras, a LED from which the intensity can be finely controlled by the LED current setting is used for illumination.³⁹ The camera readout is compared with the intensity from the LED, which is measured by a photodiode placed next to the LED. In this way, the sensitivity of the MEM-FLIM camera and the reference camera can be compared in the same optical setup.

Detection limit

The “sensitivity” of a camera can also be described by the minimum light that can be detected. When the detected signal is smaller than the noise floor of the camera, the signal will be buried in the noise. Thus the noise floor, such as readout noise and dark current noise, determines the limits of the camera sensitivity. Assuming the photon noise is Poisson distributed, the mean of the minimum signal above the noise floor ${\sigma }_n$ is ${\mu }_s$, and the standard deviation of the signal is ${\sigma }_s=\sqrt{{\mu }_s}$. We note that ${\sigma }_n^2$ is composed of several independent terms ${\sigma }_n^2={\sigma }_s^2+{\sigma }_d^2+{\sigma }_r^2+{\sigma }_q^2$.

When the integration time $T_0$ is small, the noise floor ${\sigma }_n$ is determined by the readout noise ${\sigma }_r$ of the camera. We assume that the requirement for a signal not being buried in the noise floor is that the difference between the signal level and the noise level is at least $k$ times bigger than the standard deviation of the signal, Eq. (6):

(6)

$$\begin{gathered} {\mu }_s-{\sigma }_n\ge k{\sigma }_s\Rightarrow \\ {\mu }_s-k\sqrt{{\mu }_s}\ge {\sigma }_n\Rightarrow \\ {(\sqrt{{\mu }_s}-\frac{k}{2})}^2\ge {\sigma }_n+\frac{k^2}{4}\Rightarrow \\ {\mu }_s\ge {\sigma }_n+\frac{k^2}{2}+k\sqrt{{\sigma }_n+\frac{k^2}{4}}. \end{gathered}$$

At a longer integration time, the influence of the dark current noise cannot be ignored, since the dark current noise ${\sigma }_d$ increases with the integration time $T_0$. Concurrently, the signal level is also increasing linearly with the integration time. We note that given an integration time $T_0$, the Poisson character of the photon signal and the dark current means that ${\mu }_s=v_s{T}_0$ and ${\sigma }_d=\sqrt{v_d{T}_0}$, respectively. We assume that the signal can be distinguished from the noise floor if the range of the signal does not overlap with the range of the noise, which gives us Eq. (7). Thus when the rate of electron generation ($v_s$ and $v_d$) meets the condition in Eq. (7), the signal will be above the noise floor and can be detected by the camera.

(7)

$$\begin{gathered} {\mu }_s-k{\sigma }_s\ge {\mu }_d+k{\sigma }_d+{\sigma }_r\Rightarrow \\ {v}_s{T}_0-k\sqrt{v_s{T}_0}\ge v_d{T}_0+k\sqrt{v_d{T}_0}+{\sigma }_r\Rightarrow \\ {v}_s\ge v_d+k\sqrt{\frac{v_d}{T_0}}+\frac{{\sigma }_r}{T_0}+\frac{k^2}{2T_0}+\frac{k}{\sqrt{T_0}}\sqrt{v_d+k\sqrt{\frac{v_d}{T_0}}+\frac{{\sigma }_r}{T_0}+\frac{k^2}{4T_0}}. \end{gathered}$$

It is clear from this result that for long integration time ($T\to \infty$), the signal can be detected if:

(8)

$$v_s\ge v_d+2k\sqrt{\frac{v_d}{T_0}}.$$

4.

Results and Discussion

4.1.

Camera Characteristic: Performance

4.1.1.

Linearity

A linear regression line is fitted to the intensity data for various exposure times, as shown in Fig. 4. The MEM-FLIM camera exhibits linear photometric response for almost the entire dynamic range, resulting in the coefficient of regression $>0.999995$. Since one image consists of two phase images (named phase one image and phase two image), we split these two phase images and analyze them separately.

Fig. 4

Linear photometric response of the MEM-FLIM camera.

4.1.2.

Sampling density

As shown in Fig. 5(a) and 5(b), in both the horizontal and vertical directions, the sampling densities of the MEM-FLIM camera are the same: $212\text{pixels}/170\mu \mathrm{m}\approx 1.24$ samples per micron. 170 μm corresponds to the actual dimension of the section in the stage micrometer that is scanned (Fig. 5). The MEM-FLIM camera has a square sampling. The sampling distances are $170\mathrm{\mu m}/212\approx 0.8\mu \mathrm{m}=800\mathrm{nm}$. When dividing the pixel size (17 μm) by the magnification of the objective lens ($20\times$), we get $0.85\mathrm{\mu m}/\mathrm{sample}\approx 1.18\text{samples}/\mu \mathrm{m}$. This value differs from the measured sampling density ($1.24\text{samples}/\mu \mathrm{m}$) due to internal demagnification in the microscope. The internal demagnification in the light paths of the MEM-FLIM system and the reference system are different since the light paths of the two systems are not exactly the same.

Fig. 5

Illustration of using a stage micrometer to measure the sampling density. (a), Horizontal direction view. (b), Vertical direction view. (c), Overlapping image of two phase images in (a). (d), Overlapping image of the two phase images in (b).

Both the pixel size and the pixel number in the MEM-FLIM camera are the same in the horizontal and vertical directions; however, the image has a rectangle shape. This is due to every image containing two phase images as described in Sec. 2. These two phase images can be separated. If we assign the green color to one thresholded phase image and the red color to the other thresholded phase image, by overlapping the two phase images, we see that these two phase images match very well and result in the yellow color shown in Fig. 5(c) and 5(d). Less than 2% of the pixels, as shown in Fig. 5, differ between the two thresholded phase images. The images of Fig. 5(a) and 5(b) appear stretched due to two square image pixels in the vertical direction corresponding to a single square pixel on the sensor with two storage areas.

4.1.3.

Resolution

The comparison of the OTF of MEM-FLIM and the reference camera is shown in Fig. 6. The use of the stage micrometer (as in Fig. 5) with the knowledge of the actual CCD pixel size makes it possible to determine the absolute physical frequency of cycles/mm shown in Fig. 6. The effect of differing optical magnification between the two systems is thereby compensated. The OTF of the MEM-FLIM camera is higher than that of the reference camera. As a consequence, the image quality for the MEM-FLIM camera is better than for the reference camera. Actual images will be shown later in this manuscript. The (incoherent) diffraction cutoff frequency of the lens⁴⁰ is $f_c=2\mathrm{NA}/\lambda$ which for green light ($\lambda \approx 0.5\mu \mathrm{m}$) and $\mathrm{NA}=0.6$ gives $f_c\approx 2400\text{cycles}/\mathrm{mm}$. The limiting factor in the OTF above is, therefore, not the objective lens but the camera system. The slight increase of the MEM-FLIM OTF above the objective lens OTF has two sources. First, all three curves have been normalized to unity, although the exact transmission at $f=0$ for the two cameras is probably less than 1, and second, there is a slight amount of partial coherence associated with the condensor optics.

Fig. 6

OTF comparison between the MEM-FLIM system, the reference FLIM system, and the diffraction-limited objective lens.

4.1.4.

Noise

Poisson noise distribution

The validation of the Poisson distribution of the noise source is shown in Fig. 7. The linear fit indicates that the variance of the difference images increases linearly with the mean intensity, which shows that the noise source in the image is Poisson distributed. The integration time is 180 ms.

Fig. 7

The Poisson assumption validation and the sensitivity of the MEM-FLIM camera.

Dark current noise

Figure 8(a) shows the relationship between dark current and integration time when the frame time is fixed. The mean value of each column in a dark image is calculated and plotted for different integration times. By subtracting two images obtained at the same setting, the offset and the fixed pattern of each image can be eliminated. Because dark current noise follows Poisson statistics, the variance in this difference image equals twice the average intensity in one image.⁶ The generated dark current is linear in the integration time, which is plotted in Fig. 8(b). When the integration time is 600 ms, the dark current is $76/16383\approx 0.3\%$ of the full dynamic range. Since the electron-to-ADU converting factor is known from the absolute sensitivity experiment, which is $0.43\mathrm{ADU}/{\mathrm{e}}^{-}$, the dark current can also be written as $76(\mathrm{ADU})/0.43(\mathrm{ADU}/{\mathrm{e}}^{-})/600(\mathrm{ms})=0.29{\mathrm{e}}^{-}/\mathrm{ms}$. By fixing the integration time and varying the frame time, we see in Fig. 9 that the dark current is not influenced by the frame time and can be neglected.

Fig. 8

Dark current derived from the fixed frame time of 2000 ms. (a), Relationship between dark current and integration time ($T_0$). (b), Linearity of dark current.

Fig. 9

The relationship between dark current and frame time ($T_1$) when the integration time is fixed to 100 ms. The frame time is set from 200 ms up to 2000 ms in intervals of 200 ms. The results from different frame time values are overlapped with each other.

Readout noise

Readout noise can be obtained from the fittings in Fig. 8(a). When the integration time goes to zero, the noise source due to the dark current is eliminated. Thus the constant terms in the fittings represent the readout noise. The readout noise is independent of the integration time. The average readout noise of the MEM-FLIM camera is ${\sigma }_{\text{readout}}=\mathrm{sqrt}[(34.76+34.58)/2]\approx 5.9\mathrm{ADU}\approx 14{\mathrm{e}}^{-}$. In the same way, the readout noise from the reference system can be determined to be $3.4\mathrm{ADU}\approx 6{\mathrm{e}}^{-}$ (figure not shown). The factor of 1.7 between these two results is most likely because we are working with the first version of the MEM-FLIM chip/camera, while the reference system, as an existing commercial product, is already well optimized.

4.1.5.

Sensitivity

Sensitivity

The sensitivity of the MEM-FLIM camera is shown in Fig. 7. The linear fit indicates that the noise source in the image is Poisson distributed, as explained in Sec. 4.1.4, and the slope of the fitting represents the sensitivity of the camera. The sensitivity of the MEM-FLIM camera is $0.43\pm 0.03\mathrm{ADU}/{\mathrm{e}}^{-}$. For the reference camera, the same procedure resulted in a sensitivity of $0.53\pm 0.03\mathrm{ADU}/{\mathrm{e}}^{-}$. To compare with the sensitivity of the reference camera $0.53\mathrm{ADU}/{\mathrm{e}}^{-}$, one needs to multiply the sensitivity of the MEM-FLIM camera by $2^{12-14}$, which is $0.43/4\approx 0.11\mathrm{ADU}/{\mathrm{e}}^{-}$, so that the bit differences of the two cameras (the MEM-FLIM camera 14 bits, the reference camera 12 bits) are taken into account. For these experiments, the analog gain of the MEM-FLIM camera was set to 6 dB, and the MCP voltage of the reference camera was set to 400 V.

Comparing camera sensitivities

The camera readout has a linear photometric response to the LED intensity. By fitting a straight line to the camera readout for various LED intensities, the slope of the fit indicates the ability of the camera to convert photo electrons to ADU. Sensitivity can be increased by increasing the electronic gain of the camera, as shown in Fig. 10(a). A comparison of the sensitivities of the MEM-FLIM camera to the reference camera is shown in Table 1. To increase the sensitivity of the reference camera, we can use different MCP voltages, while with the MEM-FLIM camera we can adjust the analog electronic gain. If we define the sensitivity of the reference camera at the MCP voltage of 400 V as 1, and take into consideration the bit differences of the two cameras, then we can compare the sensitivities of the two cameras. When comparing the sensitivities of the reference camera at different MCP voltages, one only needs to divide the slope of the fitting at a higher MCP voltage with the slope at the MCP voltage of 400 V (2.07). For example, the slope of the reference camera at the MCP voltage of 500 V is 8.88, which gives its sensitivity as $8.88/2.07=4.29$. For the sensitivity of the MEM-FLIM camera, one needs to convert the bit difference first by multiplying the slope of the MEM-FLIM camera by $2^{12–14}$ before dividing it by 2.07. For example, the MEM-FLIM camera at 6-dB analog gain is $24.77\times 2^{12–14}/2.07\approx 2.99$ as sensitive as the reference camera at the MCP voltage of 400 V. From Table 1 we can see that the MEM-FLIM camera ($\text{normalized slope}=59.88$) is as sensitive as the reference camera ($\text{normalized slope}=56.71$).

Fig. 10

The MEM-FLIM camera sensitivity. (a), Sensitivity increases with the increasing analog gain. (b), Sensitivity with respect to signal-to-noise ratio.

Table 1

Sensitivity comparison of the reference camera and the MEM-FLIM camera.

	Slope	Sensitivity
Reference camera 400 V	2.07	1
Reference camera 500 V	8.88	4.29
Reference camera 800 V	117.28	56.71
MEM-FLIM camera 6 dB	24.77	2.99
MEM-FLIM camera 42 dB	495.3	59.88

The cost of using a higher analog gain in the MEM-FLIM system is a reduced SNR as shown in Fig. 10(b). The SNR ($52.4:1$) in the image obtained from the MEM-FLIM camera at a small analog gain (6 dB) is higher than the SNR (20.8) in the image obtained from the reference camera at its lowest MCP voltage (400 V). For the reference camera, the SNR goes down when the MCP voltage increases. When the MEM-FLIM camera is set at a high analog gain (42 dB) and the reference camera set to a high MCP voltage (800 V), the SNR is comparable (MEM-FLIM camera: $11.7:1$; reference camera: $11.8:1$).

Dectection limit

We can get the minimum signal that can be detected by the MEM-FLIM camera from Eq. (6). When the integration time is short, the noise floor ${\sigma }_n$ will be dominated by the readout noise ${\sigma }_r$. From Figs. 7 and 8(b), we know that ${\sigma }_n={\sigma }_r=5.9\mathrm{ADU}\approx 5.9(\mathrm{ADU})/0.43(\mathrm{ADU}/{\mathrm{e}}^{-})=13.72{\mathrm{e}}^{-}$. We assume that the signal can be distinguished from the noise floor if the difference between the noise floor and the signal is $k$ times bigger than the standard deviation of the signal: ${\mu }_s-{\sigma }_n\ge k{\sigma }_s$ [Eq. (6)]. When $k=5$, based upon the Chebyshev inequality,⁴¹ the probability that the signal level can be mistakenly identified as noise will be $\le 1/k^2=4\%$. The Chebyshev inequality is distribution free, so it is not necessary to know the probability distribution of the signal. If we make use of the assumption that the signal has a Poisson distribution and that the average value of the signal is sufficiently high (${\mu }_s>10$), then the probability given above drops to $3\times {10}^{-6}$. This means signal detection at the $k=5$ level is essentially guaranteed. In this case using Eq. (6), the minimum signal that can be detected by the MEM-FLIM camera is ${\mu }_s=48.6{\mathrm{e}}^{-}$. Using the same method, the minimum signal that can be detected by the reference camera is $35.4{\mathrm{e}}^{-}$.

4.2.

Lifetime Measurement

We have measured the fluorescence lifetime of various objects, e.g., fluorescent solution and biological samples. Below are examples of the lifetime measurements on biology samples: fixed U2OS (osteosarcoma) cells that expressed GFP supplied from Leiden University Medical Center, GFP-actin–stained live HeLa cells, and GFP-H2A–stained live U2OS cells provided from the Netherlands Cancer Institute. In all experiments, the calibration is done to determine the phase and modulation change introduced by the system itself by using a fluorescein solution at 10 μM, the lifetime of which is known to be 4 ns.^31,32 The modulation frequency of the MEM-FLIM system is at this time hardwired in the MEM-FLIM camera to 25 MHz. Results from the reference system served as a basis for comparison. The typical fluorescence lifetime of GFP is 2 to 3 ns.^42,43

The fluorescence lifetime measurements are carried out in the following steps: (1) change the phase delay between the camera demodulation signal and the LED-modulated input signal and take a number of phase images—in our case, six original phase images; (2) separate two phase images from one original phase image taken in the first step (six original phase images thus produce 12 phase images), and put them in the right order; (3) correct for the background image; (4) convert the image data to the .fli format and read the file into LI-FLIM software; and (5) choose the region of interest and analyze the data.

4.2.1.

GFP-stained fixed U2OS cells

The comparative lifetime measurement was performed on the fixed GFP cell shown in Fig. 11. U2OS is a human osteosarcoma cell line. A Zeiss objective with a magnification of $20\times$ and a numerical aperture of 0.5 was used for this experiment. The integration time of the sample in both systems was set to 100 ms.

Fig. 11

Intensity and lifetime images of fixed U2OS green fluorescence protein (GFP) cells. (a)–(d) are intensity images and (e)–(h) are lifetime images. (a), Intensity image from the reference camera. (b), Magnified image of (a). (c), Intensity image from the MEM-FLIM camera. (d), Magnified image of (c). (e), Lifetime derived from the phase change for the reference camera. (f), Lifetime derived from the modulation depth change for the reference camera. (g), Lifetime derived from the phase change for the MEM-FLIM camera. (h), Lifetime derived from the modulation depth change for the MEM-FLIM camera.

To compare images from two cameras, the histograms of the two images are stretched over the range of 0 to $2^{\mathrm{BN}}-1$. One maps the intensity value $p_{\text{low}}\%$ to the value 0 and $p_{\text{high}}\%$ to by the transformation given in Eq. (2).⁴⁴ The original intensity $A$ at position $[x,y]$ then transforms to $B$. In our case, we choose $p_{\text{low}}\%$ and $p_{\text{high}}\%$ to be 5% and 99.9% to exclude the outliers. BN is chosen to be 8, so the mapped intensity range is from 0 to 255. Note that the values of $B[x,y]$ are floating point numbers.

(9)

$$B[x,y]=\{\begin{array}{ll}0& A[x,y]\le p_{\text{low}}\%\\ (2^{\mathrm{BN}}-1)·\frac{A[x,y]-p_{\text{low}}\%}{p_{\text{high}}\%-p_{\text{low}}\%}& p_{\text{low}}\%<A[x,y]<p_{\text{high}}\%\\ (2^{\mathrm{BN}}-1)& A[x,y]\ge p_{\text{high}}\%\end{array}.$$

We can see that the field of view of the reference camera is bigger than the MEM-FLIM camera in Fig. 11(a) and 11(c), but the resolution of the MEM-FLIM camera is significantly better than the reference camera in Fig. 11(b) and 11(d). Detailed structure inside the cell can be seen in the image taken with the MEM-FLIM camera. This structure is not readily visible in the image with the reference camera.

The lifetime images from the both cameras are compared in Fig. 11(e)–11(h). The MEM-FLIM camera clearly yields a better spatial resolution in the lifetime images. A $10\times 10\text{pixel}$ area was used corresponding to an area of $87{\mu \mathrm{m}}^2$ for the reference camera and $65{\mu \mathrm{m}}^2$ for the MEM-FLIM camera. The lifetimes derived from the phase change for the reference and MEM-FLIM system are $1.96\pm 0.31$ and $1.86\pm 0.48\mathrm{ns}$, respectively. The lifetimes derived from the modulation depth change are $3.05\pm 0.21$ and $3.20\pm 0.58\mathrm{ns}$, respectively. The lifetime uncertainty is the standard deviation of the 100 lifetimes in the $10\times 10\text{pixel}$ area. The modulation on the sample for the reference camera reached 0.64, while the modulation for the MEM-FLIM system is 0.55. The difference between the lifetimes derived from the phase change and the modulation change can be explained by the heterogeneity of GFP lifetime components. By doing multifrequency measurements on the reference system, the lifetime components in the sample are determined to be 1.24 ns (41%) and 5.00 ns (59%). The data are consistent with the values in the literature (1.34 ns [46%] and 4.35 ns [54%]).⁴⁵

The fluorescent lifetime, as recorded with the MEM-FLIM camera, is in good agreement with values from the reference camera. Compared to the reference camera, the lifetime uncertainties (${\sigma }^{\prime }s$) measured from the MEM-FLIM cameras are higher than those from the reference camera since the modulation depth for the MEM-FLIM camera is not (yet) as good as in the reference camera. However, image quality (detail) of the MEM-FLIM camera is significantly better than that of the reference system.

4.2.2.

GFP-actin–stained HeLa cells

For these experiments, we imaged HeLa cells, stably expressing GFP-tagged $\beta$-actin with the MEM-FLIM and reference cameras. The $\beta$-actin expression in these cells is quite low and they therefore present an example of a typical low-intensity preparation. A Zeiss oil objective with a magnification of $40\times$ and a numerical aperture of 1.3 was used for this experiment. The integration time for both the reference camera and the MEM-FLIM camera was 1000 ms. The intensity images undergo the same gray-value stretching process as described in Sec. 4.2.1.

The lifetimes derived from the phase change for the reference camera and the MEM-FLIM camera are $2.66\pm 0.49$ and $2.59\pm 0.40\mathrm{ns}$, and the lifetimes derived from the modulation depth change are $2.35\pm 0.97$ and $2.63\pm 1.46\mathrm{ns}$, respectively. The modulation on the sample for the reference system reached 1.05, while the value for the MEM-FLIM camera was 0.38. From Fig. 12, we can see that the MEM-FLIM camera has a higher resolution and a better image quality than the reference camera. The fibers in the cell can be seen in the MEM-FLIM image but not in the reference image.

Fig. 12

Intensity and lifetime images of GFP-actin–stained HeLa cells. (a)–(c) are intensity images and (d)–(f) are lifetime images (the lifetime derived from the phase change). (a) and (d), Full field of view from the reference camera. (b) and (e), Magnified region from the reference camera. (c,f), The same region from the MEM-FLIM camera.

The lifetime images derived from the phase change of both cameras are also compared in Fig. 12(d)–12(f). In the lifetime image of the MEM-FLIM camera, the difference within the cell—the spatial variation—can be seen. Just above the middle of the image the lifetime (color) differs from the surrounding cellular material (as shown within the white rectangle). This structure can also be seen in the intensity image. This detail is blurred in the lifetime image from the reference camera.

4.2.3.

GFP-H2A–stained live U2OS cells

For these experiments, we imaged U2OS cells, stably expressing GFP-H2A with the MEM-FLIM and reference cameras. A Zeiss oil objective with a magnification of $40\times$ and a numerical aperture of 1.3 was used for this experiment. The image comparison in Fig. 13 again shows that the MEM-FLIM camera has a higher resolution than the reference camera, while the reference camera has a larger field of view than the MEM-FLIM camera. The integration time for both the reference camera and the MEM-FLIM camera was 200 ms, and the phase-based, lifetime results are comparable with $2.65\pm 0.48\mathrm{ns}$ measured by the MEM-FLIM camera and $2.57\pm 0.20\mathrm{ns}$ measured by the reference system. The intensity images undergo the same gray-value stretching process as described in Sec. 4.2.1.

Fig. 13

Intensity images of live U2OS cells. (a), Full field of view from the reference camera. (b), Magnified image of a region from the reference camera. (c), The same region from the MEM-FLIM camera.

5.

Discussion and Conclusion

We have designed, built and tested an all-solid-state CCD-based image sensor and camera for fluorescence lifetime imaging. A detailed comparison between the MEM-FLIM and reference cameras is shown in Table 2. Using the MEM-FLIM camera, we successfully measured the lifetimes for various fluorescent objects including biological samples.

Table 2

Comparison of the MEM-FLIM and the reference cameras.

a

The pixel size of the CCD sensor itself is 6.45 μm; we are using 2×2 binned mode, which gives 12.9 μm, and the pixels as “projected” onto the photocathode by the fiber optic taper are magnified 1.6×, arriving at 20.6 μm of effective pixel size of the intensified camera system.

The MEM-FLIM results are comparable to the reference system. There are several advantages for the MEM-FLIM system over the reference system. (1) The camera can be modulated at the pixel level, permitting the recording of two phase images at once. The acquisition time can thus be shortened by using the MEM-FLIM camera, which causes less photobleaching in the biological sample. (2) The MEM-FLIM camera does not need high-voltage sources and RF amplifiers, and the system is more compact than the reference system. (3) In the MEM-FLIM system, one can change the integration time and the analog gain, which has no effect on the optical system itself. In the conventional frequency-domain FLIM system, one needs to control both the integration time and the MCP voltage to make use of the full dynamic range of the camera. However, changing the MCP voltage by more than approximately 50 V (depending on the intensifier and the MCP voltages used) means changing the system itself, which in turn means that the calibration done at another MCP voltage is no longer reliable. So one needs to pay extra attention when adjusting the settings on the conventional frequency-domain FLIM system. (4) Possible sources of noise and geometric distortion are significantly reduced. (5) The image quality from the MEM-FLIM camera is much better than the conventional intensifier-based CCD camera, and the MEM-FLIM camera thereby reveals more detailed structures in the biological samples. (6) The quantum efficiency of the MEM-FLIM camera is much higher than the reference camera. For the MEM-FLIM camera, the quantum efficiency is determined by the characteristics of the front-illuminated CCD, about 30%, 50%, and 70% at 500, 600, and 700 nm, respectively. For the reference camera, the quantum efficiency of the photo cathode at 500 nm is around 11%. Further, there are losses in other parts of the system, including the fiber optics and the CCD camera, not all of which can be attributed to true quantum effects.

It is also interesting to compare our results to the previously developed CCD camera described in Refs. 26 and 27, as shown in Table 3. Both the SR-2 and the MEM-FLIM cameras are able to measure fluorescence lifetimes, and the modulation depth and the lifetime results are comparable. The quantum efficiencies of the two cameras are comparable since they are both determined by the characteristics of a front-illuminated CCD. There are big improvements in the MEM-FLIM camera compared with the SR-2 camera. Although both the MEM-FLIM and the SR-2 cameras are noncooled, we can see clear influence of the dark current on the SR-2 camera. The presence of an edge artifact in the phase images in Fig. 2(e) and (2f) of Ref. 26 and Fig. 3 of Ref. 27 can be attributed to the dark current. In the MEM-FLIM camera, however, there is a uniform phase response across the sensor and the dark current influence can be ignored. The MEM-FLIM camera has more than twice as many pixels, smaller pixel sizes for better spatial sampling density, and a fill factor that is 2.75 times that of the SR-2. The modulation frequency of the MEM-FLIM camera described in this manuscript is 25 MHz, while the SR-2 camera is 20 MHz. As mentioned in Refs. 26 and 27, the modulation frequency can, in principle, be significantly increased for both cameras, but all measurements of camera performance would have to be re-evaluated for any higher frequency. At this time we can only compare performance at the frequencies that have been used.

Table 3

Comparison of the MEM-FLIM camera and the SR-2 camera.

Besides transferring the photo-generated charge alternately to the two adjacent CCD storage registers in the vertical direction at the modulation frequency (as shown in Fig. 2), we also tried another architecture: transferring the charge to registers located in the horizontal direction. The former architecture is called vertical toggling, while the latter is called horizontal toggling. The architecture of the horizontal toggling sensor is similar to an interline CCD. The advantage of horizontal toggling design over the vertical one is that in the vertical toggling design the light source needs to be switched off during the image transfer period, since the photogate of the sensor is also used for charge transfer. In the horizontal design, however, dedicated registers are used to transfer the charge, which means there will be no smear effect if the light is left on during image transfer. Since this disadvantage of vertical design can be overcome by using a properly designed light source and the vertical toggling design outperformed the horizontal design (data not shown), we focused on the vertical toggling design as the architecture of choice for the system.

6.

Future Work

The MEM-FLIM camera is able to measure the fluorescence lifetime, but the modulation frequency is now limited to 25 MHz. We intend to achieve higher modulation frequencies in the next-generation camera. The next-generation camera will also have larger pixels (better light gathering) and more pixels (larger field-of-view) compared to the current design. Improved chip-level mask design should improve the modulation depth.

The camera is not perfect and there is still room for improvement. For example, the lifetime derived from the phase change is quite stable, but when the integration time of the experiment is increased, the lifetime derived from the modulation depth change has a tendency to increase as well. This effect can be explained by a known defect in this version of the MEM-FLIM sensor chip. The MEM-FLIM chip has a mask protecting parts of the surface from exposure to photons. In the current version, there is a slight displacement of the mask from its intended position. This means that the photoelectrons that we measure are to a certain extent caused by contributions from the wrong source. This defect will be corrected in the next version of the sensor chip.