2.1.

Related Work and Challenges

An integrated imaging-rendering system has been developed using field-programmable gate array,^1,10,11 which enables automated acquisition with LSCEM and, subsequently, real-time volume rendering of mucosal tissue. However, rendered results from the system clearly exhibit alignment problems, which indicate a need for complementing registration methods.

Intensity-based registration is a class of registration algorithms that does not require parametric transformations.¹² These algorithms produce a dense array of translation vectors for each voxel, indicating its desired deformation. Intensity-based methods are not feature dependent, and do not require a preliminary feature extraction step. The Demons algorithm⁹ (henceforth referred to as Demons) is a prominent intensity-based, nonrigid registration algorithm, which registers nonlinear image transformations by modeling the deformations as a rapidly diffusing process. This method is well known for its effectiveness and fast convergence⁶ and is suitable for a wide range of medical applications.¹³

Rendering of real-time deformable volumes has been developed, notably for texture-based methods.^14–16 However, texture mapping algorithms require a complete, static dataset in order to function efficiently, where a proxy geometry that speeds up the rendering process is often created in the prerendering stage. In this case, proxy generation is not feasible because the dataset cannot be readily used in real time.

Object-ordered rendering methods such as splatting¹⁷ iterate through volume voxels and project them onto the viewing screen. Thus, rendering volumes with dynamic sizes is intuitively straightforward. On the other hand, image-ordered ray-casting¹⁸ rendering is also able to render volumes of dynamic sizes. This is achievable by dynamically updating render parameters, including dataset thickness.

Well-known functional deformation methods such as spatial ray deflectors¹⁹ bend the rays within a 3-D space in the opposite direction of the deformation. Ray-casting rendering of free-form deformation (FFD) volumes known as inverse ray deformation²⁰ has been presented using B-spline functions. This method bends the ray paths instead of the volume, bypassing the need for an intermediate deformed volume. Recently developed methods specially targeted at the graphics hardware pipeline^15,21 deform images using parameterized functions. However, the computational costs of FFD methods increase exponentially with a larger amount of overlapping deformation functions. Especially, dense biomedical datasets such as living tissue require sufficiently complex parametric deformation models to achieve high accuracy, and this impairs computational performance.

In this paper, we replace the functional deformations with displacement vectors computed using the Demons algorithm. We focus on the coupling between the registration and visualization processes. Our method omits the reconstruction stage in the main real-time operational flow between these processes by injecting the registration transformation through a vantage opening within the volume rendering pipeline.

2.2.

Image Registration with Demons

Our registration model is specifically designed for the registration of slice images acquired from the LSCEM system. We assume that distortions occur within each slice; i.e., sheared motion is exhibited in a 2-D plane parallel to the imaged plane. Also, due to the small physical distance ($4\mu \mathrm{m}$) between consecutive slices,² adjacent slices can be assumed to have a high correlation where unwanted motion is relatively small. With these notions, each slice can be registered against the previously transformed slice to an appropriate tolerance level.

In general, the dataset is represented as an isotropic volume set $V$, which is obtained by sampling data points of regularly spaced intervals. A sequence of 2-D planes perpendicular to the imaging probe direction is captured across incrementing depths, forming a volume. A consolidation of $k$ number of consecutive slices forms a 3-D dataset: $V=\{I_k|k=1,\dots ,k_{\max },k\in Z^{+}\}$. In order to realign these slices, the transformation is a deformation within each individual slice, i.e., interslice registration. Here, two image slices are involved: a reference (fixed) image $I_F$ and a target (moving) image $I_M$. With a 2-D transformation denoted by ${\mathbf{T}}_k[U_x,U_y]$, where $U_x$ and $U_y$ are matrices representing displacements in the $x$- and $y$-direction, the registration model is

2.3.

Demons Displacement

Based on the optical flow model, the Demons algorithm⁹ computes a displacement vector that transforms $M$ as closely as possible to match $F$. Our approach is straightforward: continuously deforming $M$ with an incremental transformation, which minimizes the energy difference between the transformed image and reference image.

An accelerated variant²³ of Demons is selected, such that the displacement vector is not bounded solely by the fixed image gradient $\nabla {\mathrm{I}}_F$, where $\nabla$ is the gradient operator. This variant aims to mitigate the inefficiency experienced by a small fixed image gradient. Given the accumulated displacement across iterations $t$ to be ${\mathbf{T}}_t={\mathbf{T}}_{t-1}+\mathbf{u}$ and $I_{M_t}=I_{M_0}\circ {\mathbf{T}}_{t-1}$, the accelerated Demons displacement $\mathbf{u}$ is

2.4.

Regularization, Optimization, and Transformation

Estimating the nonrigid deformation between matching image pairs is an ill-posed problem. For instance, all points with the same intensity value in the moving image can theoretically be mapped into a same point in the fixed image with an identical value. This produces a solution with a high similarity metric score, despite being inaccurate.

To solve the problem, additional spatial constraints must be incurred. We use a regularization process to alleviate the ill-posedness and bound local transformations together by relating neighborhood displacements. An analogy of this effect will be a force exerted on a point within the volume that should also displace its neighborhood to some extent. It has been deduced that regularization plays a crucial role in determining the accuracy and robustness of nonrigid registration.²² The choice of the kernel is dependent on the type of dataset and the anticipated transformation. The Gaussian low-pass filter is chosen in the regularization operation as

The terminating criterion for each slice is subject to either one of two factors: (i) as soon as the transformation count exceeds a predefined number of iterations or (ii) when the slices match each other closely enough according to the designated similarity measure, fulfilling Eq. (2).

Our proposed registration method has a unique characteristic in which the deformation model is readily embedded within the regularization of the Demons displacements. Thus, the deformation profile is represented as an array of point-displacement vectors without further modeling. As compared with inverse-ray-deformation methods²⁰ or parametric methods,^12,24 functional deformation kernels or additional modeling constraints are not required. Therefore, in our ray-deformation model, sampled points are displaced by $\mathbf{T}$.

3.1.

Object-Ordered Rendering

In object-ordered rendering, a projection of each voxel on the viewing screen is first computed. The voxel values are then composited with the screen pixels at the projected position. For a detailed description, the reader can refer to Ref. 17.

In our proposed design, we compute the corresponding screen projections while iterating through the voxel locations. To incorporate deformation, the voxel values that are composited with screen pixels are obtained by resampling the voxel location $S$ displaced by $\mathbf{T}$ [see Fig. 4(a)]

Algorithm 1 illustrates this process.

Algorithm 1

Object-ordered deformable rendering.

3.2.

Image-Ordered Rendering

Image-ordered rendering, or ray-casting, casts imaginary rays toward the data object and samples points along the ray. Sampled points on a common ray are combined together to obtain a rendered image. In our proposed design, we deflect the sampling points along each ray with deformable forces.

We draw distinctions from functional deformation methods¹⁹ and FFD models.²⁰ Our method does not deflect the ray path itself directly; rather, we cast the ray that samples the dense transformation matrix $\mathbf{T}$. Then, the displacements are combined with the spatial coordinates to indicate the voxel position to be sampled in the captured dataset.

Given the center of projection of the scene $C_{\mathrm{proj}}$, from the ray origin on the screen $P$, a ray is cast in the direction

The sampling point coordinates $\mathbf{S}$ are thus

The transformation $\mathbf{T}$ is then sampled at $S$ using an interpolation function to obtain the displacement of sampled point $\mathbf{T}(S)$, and the sampled voxel intensity is

Algorithm 2 illustrates this process.

Algorithm 2

Image-ordered deformable rendering.

There are two sampling stages that are involved in the image-ordered deformable rendering method. First, the deformation matrix is sampled (step 7), which gives the resultant displacement vectors of the sampled points. Next, the dataset is sampled at the displaced locations (step 8).

5.1.

Real-Time Rendering of Demons Deformable Datasets

To demonstrate image registration, a swine tongue dataset captured with confocal microscopy is used. The full dataset is 19 slices thick with a resolution of $1024\times 1024$. Synthetic smooth deformation generated using a spherical filter is applied to this dataset to obtain a deformed dataset. The original nondeformed dataset serves as the ground truth. The datasets are registered and visualized with MIP in our experiments, as shown in Fig. 9 as renderings at increasing slice counts.

Fig. 9

Registration–rendering of the swine tongue dataset using MIP, (a–b) full datasets: (a) original (fixed) image, (b) synthetically deformed (moving) image; (c) the synthetic deformation; (d–g) rendered at different thickness counts of (a) 2, (b) 8, (c) 14, and (d) 19.

This experiment simulates nonrigid deformations and the use of our approach to obtain visual hints about the captured dataset. It can be observed that registration is performed to realign structures in the tissue, and cross-sectional information can be realized by visually perceiving the renderings.

Experiments on a living tissue dataset of a Drosophila muscle obtained from in vivo experiments²⁵ are also performed. In this experiment, two datasets of the same tissue captured at different times are used. This sufficiently indicates natural deformations of living tissue. They contain 29 slices with a resolution of $1024\times 1024$ across $0.67\times 0.67\times 2.2\mu {\mathrm{m}}^3$. The fixed and moving images are recorded 60 min apart.

Figure 10 shows registration–rendering results using the averaging compositing scheme. This experiment demonstrates the registration and visualization of natural deformations in living tissue. There are two different conditions under which this real-time registration–visualization pipeline is useful: (i) realignment of slices against each other due to prolonged capture time, which is a limitation of modern imaging modalities;^1,5 and (ii) registration against another predefined atlas dataset, such as between time-lapse datasets or toward a dataset with well-established features so that the current acquisition is coherent.

Fig. 10

Registration–rendering of the Drosophila muscle cytoplasm dataset using averaged compositing, (a–b) full datasets: (a) fixed image captured at T1, (b) moving image captured at T2; (c–f) rendered at different thickness counts of: (c) 5, (d) 13, (e) 21, and (f) 29.

5.2.

Visualization of Deformations for Physical Correction

In this experiment, we show the use of visualizing Demons displacement profiles for analyses under actual circumstances, The Drosophila muscle datasets captured at different points of time are shown in Fig. 11. The deformities expressed by these datasets are natural deformations due to biological functions and motion; no synthetic alterations are imposed.

Fig. 11

Visualization of transformations and renderings of a Drosophila muscle dataset deformed due to time-lapse motion captured at $t=0$ and $t=60\min$, intensity threshold of [80, 255]. Renderings: top left: fixed image; bottom left: moving image; top right: transformation magnitude image; bottom right: Demons-registered image. Transformation profiles are visualized in the middle.

In this experiment, we assume the dataset captured at $t=0$ to be our ground truth dataset, whereas the $t=60\min$ dataset is assumed to be the live in vivo dataset. Due to naturally occurring biological functions, i.e., metamorphosis in this case, the captured dataset at $t=60\min$ exhibits nonrigid motion at localized areas within the slices. In applications with large time gaps between acquired images, the displacement information should be captured and correction is undesirable. In these cases, the visualization of these deformities is useful for providing a clear indication of such movements for analyses where areas with higher displacements can be observed, as shown in Fig. 11.

Finally, to highlight the significance of rendering deformations for correction and realignment, we present distinctions between a default unaltered dataset and one that is Demons registered. Figure 12 shows renderings of a naturally deformed Drosophila cell nuclei dataset. Renderings are shown at different dataset thicknesses during acquisition. It is observable that without registration, the movement of cells quickly dissolves the information at an early acquisition stage. Registration of the dataset is performed by matching each newly acquired slice with an adjacent registered slice captured at the previous depth level. Only the first slice is unaltered. With registration, the effects of movements are mitigated, resulting in a more apprehendable visualization.

Fig. 12

Registration–rendering of a Drosophila cell nuclei dataset without registration (top row) and with registration (bottom row) at different slice thicknesses: (a) 5, (b) 13, (c) 21, and (d) 29.