Computational multifocus fluorescence microscopy for three-dimensional visualization of multicellular tumor spheroids

Abstract. Significance: Three-dimensional (3D) visualization of multicellular tumor spheroids (MCTS) in fluorescence microscopy can rapidly provide qualitative morphological information about the architecture of these cellular aggregates, which can recapitulate key aspects of their in vivo counterpart. Aim: The present work is aimed at overcoming the shallow depth-of-field (DoF) limitation in fluorescence microscopy while achieving 3D visualization of thick biological samples under study. Approach: A custom-built fluorescence microscope with an electrically focus-tunable lens was developed to optically sweep in-depth the structure of MCTS. Acquired multifocus stacks were combined by means of postprocessing algorithms performed in the Fourier domain. Results: Images with relevant characteristics as extended DoF, stereoscopic pairs as well as reconstructed viewpoints of MCTS were obtained without segmentation of the focused regions or estimation of the depth map. The reconstructed images allowed us to observe the 3D morphology of cell aggregates. Conclusions: Computational multifocus fluorescence microscopy can provide 3D visualization in MCTS. This tool is a promising development in assessing the morphological structure of different cellular aggregates while preserving a robust yet simple optical setup.


Introduction
Three-dimensional (3D) culture of cancer cells mimics the in vivo microenvironment more closely compared to two-dimensional (2D) monolayer cell culture (e.g., in a petri dish). In this regard, imaging of cell aggregates known as 3D multicellular tumor spheroids (MCTS) is of high relevance. 1 MCTS recapitulate key parameters of the tumor microenvironment, such as gradients of hypoxia and extracellular pH, which makes them a more realistic model of the early tumor environment than the standard methodology of 2D cell culture. 2 Since most cellular components are colorless, to observe for example the nuclei in MCTS, cells are usually stained with DNA binding fluorescent probes such as 4',6-diamidino-2-phenylindole (DAPI) and observed through a fluorescence microscope. 3 Fluorescent staining of DNA by DAPI then allows to visualize of cell nuclei within 3D MCTS 4 to provide morphological information about the architecture of the MCTS. 5 However, limited depth-of-field (DoF) emerges as an optical limitation which makes it impossible for the all-in-focus visualization of the 3D structure of a thick sample in a single image. One way to achieve 3D fluorescence imaging is by means of optical sectioning in confocal, 6 in structured illumination, 7 or in light-sheet microscopy 8 at the cost of rather complex optical setup and calibration. Digital holography has also been proposed for fluorescence microscopy 9 to retrieve 3D information by incorporating a reference beam into the setup with a consequent extra alignment in the system. Transport of irradiance equation 10 is an alternative, noninterferometric technique, where phase distribution needs also to be retrieved from defocused fluorescence images to estimate, after inverse Fresnel propagation, focused images at different planes. Other methods are based on acquiring spatially multiplexed information from the sample. This is the case for light-field microscopy where a microlens array is inserted in front of the microscope's image sensor to simultaneously capture 2D spatial and 2D angular information, [11][12][13] integral imaging, 14,15 plenoptic projection fluorescence tomography, 16 or 3D autocorrelation reconstruction in combination with phase retrieval tomography. 17 Also a diffuser in the pupil plane consisting of randomly placed microlenses with varying focal lengths has been implemented; in this case, the random positions provide a larger field of view compared to a conventional microlens array, and the diverse focal lengths improve the axial depth range. 18 Another interesting approach is Fourier ptychographic microscopy, which iteratively stitches together a number of different angles illuminated, low-resolution intensity images in Fourier space to produce a wide-field, high-resolution complex sample image. 19,20 On the other hand, multifocus (focus-stacking or z-stacking) microscopy 21,22 is a simple technique where a scanning mechanism is introduced into a wide-field microscope in order to allow the acquisition of a set of differently focused images along the optical axis. Extended DoF (or allin-focus) image is usually recovered using focus-recognition algorithms and depth-map retrieval.
As a way to overcome DoF limitation while achieving 3D visualization in fluorescence microscopy, in the present paper we propose a method based on multifocus sensing where a custom-built fluorescence microscope incorporating an electrically focus-tunable lens (EFTL) is employed to optically sweep in-depth the structure of MCTS. The EFTL allows a nonmechanical scanning in order to avoid lateral displacements between acquired images (neither the sample nor the optics are moved) 23,24 and once multifocus images are taken, image registration is performed to match the different fields of view. Then a Fourier domain post-processing approach 25,26 -which does not require depth-map estimation or segmentation of in-focus regions-is applied and the acquired information is reorganized through algorithms to allow DoF extension, synthesis of novel viewpoints as well as reconstruction of stereoscopic pairs which can serve as 3D visualization tools of a thick biological sample. Validation experiments corresponding to 3D visualization of MCTS are presented.
The biological sample used in this work is from human prostatic carcinoma cell line (LNCaP) 27 and was cultivated to form MCTS by means of the hanging drop method in which cells are suspended in droplets of medium where they develop into coherent 3D aggregates and are readily accessed for analysis. 28 Cells were then stained with DAPI with a broadband excitation centered at 358 nm and emission at 461 nm. An extra filter centered at 457 nm and bandwidth 22 nm was placed before the sensor in order to enhance the contrast of the images.
Parts of the sample to be captured in-focus are those placed at the conjugate image plane of the sensor, which is shifted from the working distance plane of the microscope objective by an amount z given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 1 ; 1 1 6 ; 3 7 5 (1) where f eq is the equivalent focal length of the combination of the microscope objective and the EFTL and verifies E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 2 ; 1 1 6 ; 3 0 3 while D is the distance between the EFTL and the sensor and d the distance between the back principal plane of the microscope objective and the EFTL. Optical power P of the EFTL can be varied between 3 and −2 diopters for currents between 270 and −230 mA. Since in our setup D ≈ 10 cm, d ≈ 5 cm, the maximum focusing (or z) range of the system is ∼210 μm.
In-focus parts of the sample are obtained at the sensor with lateral magnification M given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 3 ; 1 1 6 ; 2 1 1 which varies along the focusing range with a maximum relative change of ∼15%. This change is reflected in turn in a change in the FoV along the images obtained while the current through the EFTL varies. The multifocus stack was acquired for a set of currents in the EFTL j k ; k ¼ 1; : : : ; N between 265 and 125 mA in steps of −10 mA. The N ¼ 15 image stack is shown in Fig. 2. Note that field of view is not constant along with the stack of images and needs to be corrected before the synthesis of novel viewpoints from multifocus stack is performed. 29 Since in the present work the EFTL is positioned in the set-up in a way that total intensity remains constant between the acquired images (illumination path is not affected by the change in focus of the EFTL) this allows us to use conservation of energy (i.e., integral of intensity values in a given image of the stack should be constant) to implement registration between images of the stack (note that in other works including an EFTL, 24 illumination intensity changes between the acquired images due to the position of the EFTL in the set-up, so conservation of radiant energy does not hold and registration needs to be performed following different approaches). Energy is evaluated for a given reference image (in our case k ¼ 1 image) and the rest of the images in the stack are rescaled to give the same value. Then the captured visual information is reorganized through a Fourier domain postprocessing approach which does not require depth-map estimation or segmentation of in-focus regions. Image reconstruction is accomplished considering only, besides an effective parameter, the current through the EFTL for each image in the stack.

Image Formation Model and Novel Viewpoint Synthesis
Once the multifocus image stack is acquired, postcapture processing algorithms enable the synthesis of images with novel viewpoints of the scene. 25 Let i k be the intensity distribution of the k'th image of a stack of N images. [For color images in RGB space where f k is the in-focus region of i k . The part of the scene that is out-of-focus in i k comes from the 2D convolution between f k 0 (in-focus part of i k 0 ) and the 2D intensity PSF h kk 0 ðx; yÞ associated with the currents j k and j k 0 E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 5 ; 1 1 6 ; 6 3 2 where E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 6 ; 1 1 6 ; 5 7 1 and E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 0 7 ; 1 1 6 ; 5 1 8 where R is the aperture of the imaging system, α is the linear coefficient for the relation between lateral magnification and current through EFTL and p is the pixel pitch of the camera. For the stack of images in Fig. 2 If Hðu; vÞ is invertible, then the solution to the linear system given by Eq. (8) isFðu; vÞ ¼ H −1 ðu; vÞĨðu; vÞ, but if Hðu; vÞ is not invertible (as for the DC frequency components), then a solution to the system may be found through the Moore-Penrose pseudoinverse H † . 30 The Moore-Penrose pseudoinverse provides the set of vectors that minimize the Euclidean norm kHðu; vÞFðu; vÞ −Ĩðu; vÞk in the least squares sense. Thus, the minimal norm vector is given as The reconstruction of an arbitrary horizontal viewpoint of the scene is accomplished by simulating the displacement b x of a pinhole camera in the horizontal direction with respect to the center of the original pupil (similarly for a b y displacement in the vertical direction). The horizontal disparity d k between the images of a given point of the in-focus component f k as seen by the sensor of a centered pinhole camera and a pinhole camera displaced to the left is given as E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 1 ; 1 1 6 ; 6 9 9 d k ¼ b x αj k ; (11) (aside from a constant factor independent of k and related to the magnification at zero current). Then, in the piecewise planar approximation of the 3D scene, to obtain a shifted viewpoint s b x ðx; yÞ, each focus slice f k ðx; yÞ should be shifted in an amount according to the disparity associated with the j k current through EFTL and the baseline displacement ðb x Þ of the camera E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 2 ; 1 1 6 ; 6 1 8 In particular, s 0 ðx; yÞ recovers the image as captured with a pinhole camera in the center of the original circular pupil (i.e., extended-DoF or all-in-focus image reconstruction of the scene 23 ).
By means of the FT shift theorem, which states that translation in the space domain introduces a linear phase shift in the frequency domain, 31 In order to achieve visualization with full parallax, it is straightforward to extend Eq. (15) to the case of vertical motion simulation E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 7 ; 1 1 6 ; 2 3 0 S β y ðu; vÞ ¼ X N k¼1 e −j2πj k β y R 0 ðpvÞ ðH † ðu; vÞĨðu; vÞÞ k ; (17) and consider the synthesis of new scene perspective as seen from a pinhole camera, translated a fraction β y upward of the center of the original circular pupil by Fourier inverse transform of Eq. (17) E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 8 ; 1 1 6 ; 1 4 3 The proposed method is then able to reconstruct the extended DoF and allows the visualization of the reconstructed scene from different perspectives without previous segmentation of the focused regions from the images in the stack. However, it is possible to retrieve the depth map by combining this method with other schemes. 32 3 Results

Extend Depth-of-Field and Viewpoint Synthesis
Reconstruction of the extended DoF or all-in-focus image corresponds to β x ¼ 0, (i.e., as seen with a centered pinhole camera) and is shown in Fig. 3. Note how, unlike original images of the stack in Fig. 2, individual cell nuclei from different depths of the aggregate can be clearly seen in a single image.
If we instead consider arbitrary fractional displacements β x , β y , the corresponding viewpoints can be synthesized from Eqs. (16) and (18), respectively (combination of horizontal and vertical viewpoints is straightforward). A complete set of novel viewpoints for −0.25 ≤ β x ≤ 0.25, −0.25 ≤ β y ≤ 0.25 is available in Video 2.

Stereoscopic Pairs for 3D Visualization
Binocular vision is based on the fact that 3D objects are perceived from two different perspectives due to the horizontal separation between our left and right eyes. As a result, the left and right images of a 3D scene in our retinas are slightly different. This retinal disparity between the images provides the observer with information about the relative distances and depth structure of 3D objects. Both perspectives of the same 3D scene are fused by the brain to give the perception of depth. 33,34   In a similar way, a pair of stereoscopic images can be generated by considering a virtual stereocamera 35 formed by a left pinhole camera displaced to the left of the center of the original pupil, b x ¼ B∕2, and a right pinhole camera displaced to the right of the center of the original pupil, b x ¼ −B∕2, where the separation B between the left and right virtual pinhole cameras is known as the baseline. Since points of view from outside of the aperture have no physical meaning, B ≤ 2R. Then, it is straightforward to reconstruct the left and right views according to E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 1 9 ; 1 1 6 ; 5 9 6 i L ðx; yÞ ¼ s B∕2R ðx; yÞ; E Q -T A R G E T ; t e m p : i n t r a l i n k -; e 0 2 0 ; 1 1 6 ; 5 5 2 where each r.h.s. is to be calculated by means of Eq. (16). Once the stereoscopic pair is generated, the left and right images can be displayed in different ways. 36 In Fig. 4, the cross-eye stereo pair for B ¼ R∕2 is presented. With some practice, the fused image is perceived by deliberately crossing one's eyes until the two images come together.

Performance Assessment
Quantitative comparison can be performed with the help of a synthetic multifocus stack since, unlike the real stack, a ground-truth reference for each point of view of interest can be constructed. Figure 5(a) shows the synthetic 3D scene representing three rings of fluorescent beads with each ring lying on a different plane at distances z i ; i ¼ 1;2; 3. Images of the stack corresponding to the system focusing on each of these planes (for currents j i ; i ¼ 1;2; 3) are shown in Fig. 5(b 1−3 ), respectively. Figure 5(c 1−3 ) shows the ground truth for the scene as viewed from a pinhole camera displaced to the left, center, and right, for relative displacements β x ¼ 0.5; 0; −0.5, respectively. The multifocus stack of Fig. 5(b 1−3 ) is used to render the same viewpoints and the results obtained by means of Eq. (16) are presented in Fig. 5(d 1−3 ), respectively. Table 1 shows the mean square error resulting from the comparison (of luminances) against the ground truth for each relative displacement, showing a very good agreement between the reconstructed viewpoint and its corresponding ground truth.

Conclusion
We have developed a custom-built fluorescence microscope that incorporates an electrically focus-tunable lens and allows us to acquire sets of multifocus images from thick biological samples, in particular MCTS.
Our algorithms operated then over the acquired stacks to accomplish extended DoF by multifocus image fusion without depth-map estimation or segmentation of the in-focus regions.
Besides all in focus reconstruction along the optical axis, viewpoint synthesis with shifts in perspective can be performed to provide a stereoscopic pair of images of the sample as well as 3D visualization of the 3D structure of the cell aggregates.
Our proof-of-principle experimental results show the potential of the present approach, which could serve in a wide range of biological and biomedical applications where 3D visualization of a biological sample might be useful. As a future line of work, it might be interesting to include more fluorescent channels to assess different cellular structures.

Disclosures
The authors have no conflicts of interest to disclose.