2.1.

Implementation of the Multispectral Imaging System

A schematic of the MSI platform is presented in Fig. 2. A predetermined narrowband high-power light-emitting diode (LED) light source (SR-02, Quadica Developments Inc., Ontario, Canada) was used with a fiber-optic ring light guide and a condensing lens to generate three uniform illumination lights (center wavelengths 470, 600, and 770 nm) in series. These wavelengths in the visible spectrum were selected to demonstrate hemoglobin absorption and to examine the effects of wavelengths on penetration depth.²⁴ Since the LED light was unpolarized and the use of nonpolarization-maintaining fibers randomized the polarization state of the light,²⁵ we applied a polarizing sheet onto the distal end of the ring light guide to create linearly polarized illumination. Reflected light from the tissue passes back through the empty space of a ring light guide, a rotating linear polarizer filter (46 mm, Prinz Optics GmbH, Stromberg, Germany), a macrolens (Fujinon HF 12.5 SA-1, Phoenix Imaging Ltd., Michigan), and finally reaches a near-infrared camera (acA2000-50gmNIR, Basler, Pennsylvania). By adjusting the angle of the linear polarizer attached to the camera, we can effectively control the amount of polarization effects. To reduce specular reflections from the tissue surfaces, two linear polarizers were set orthogonal to each other. Three different spectral images were acquired at 6 fps, with the image size of $1280\times 1080\text{pixels}$. LED-based MSI has an advantage over hyperspectral imaging in that it enables high-speed image acquisition and data processing, which could be potentially useful for real-time guidance. Both LED control and image acquisition were programmed using a custom C# script (Visual Studio 2010, Microsoft). The acquired images were cropped to obtain the tissue region of interest ($735\times 637\text{pixels}$) for image processing. The 470-nm band images were selected for blood vessel segmentation.²⁶ In addition, all three spectral band images were combined to form composite images that were used for the multispectral analysis.

Fig. 2

(a) Schematic of the multispectral imaging (MSI) system and (b) photo of system implementation.

2.2.

Animal Tissue Preparation

Fresh porcine small bowels were obtained from a local abattoir and dissected into segments of 20 to 30 cm long. The sample was moistened with physiological saline and preserved at 4°C for up to 30 h from the time of slaughter until imaging. Before imaging, different segments of the small and large bowels were dissected into 5-cm-long specimens. During the measurements, the remaining samples were preserved in saline in sealed sterile containers for hydration maintenance for up to 30 min.

2.3.

Image Analysis and Suture Map

In this study, three tissue maps of blood vessel segmentation ($V$), thickness differentiation ($T$), and bite depth based on multispectral tissue classification ($B$) were produced.

The combination of the above three mentioned tissue maps resulted in the suture map. The intensity levels of the suture map, $J(u,v)$, range from 0 to 1, where 0 denotes a location that should absolutely be avoided for suture and 1 denotes a location that could be used for suture with minimal complications. The fusion operator used to combine these different matrix maps into a single suture map is the elementwise matrix multiplication:

The optimal suture points could be calculated automatically by solving the following optimization problem locally:

This optimization problem is not convex and does not have a global maximum. Local maxima were extracted as candidates for suture placements. A computationally efficient method to solve this nonlinear optimization problem approximately is to first eliminate all the pixels that are smaller than a threshold. The remaining pixel values are then compared with their eight neighbors. If a value is larger than all eight neighbors, then it is kept as a local maximum. This method will find most of the local maxima in the suture map image quickly. Using an equidistance consistency constraint, the candidate list of placements can be refined to include only equidistant suture placement recommendations. The final list of recommendations along with a colormap visualization of the suture map $J$ is provided to the surgeon to make informed decisions on avoiding vessels, choosing thick tissue to retain stronger forces, and be at an accepted distance from the lumen cut section.

3.1.

Multispectral Imaging

Figure 3 shows a porcine intestinal tissue imaging result at three different wavelengths, where the superficial features, including the blood vessels, are accentuated at 470 nm (red arrows). At 770 nm, light penetrates deeper within the tissue, revealing subsurface features [yellow arrows in Figs. 3(c) and 3(f)]. This figure also demonstrates that the cross-polarization scheme can successfully eliminate surface reflections such as glare from the tissue. Note that as the illumination wavelength band increases from blue to red and near-infrared, the image contrast decreases owing to the increased scattering and mean free path at longer wavelengths.

Fig. 3

Different spectral band images of a tissue sample with (a)–(c) and without cross-polarization, (d)–(f) demonstrating surface reflection removal. Arrows indicate features of blood vessels in red color (a) and (d) and the revealed subsurface features in yellow color. (c) and (f) White scale bars: 2 mm.

3.2.

Blood Vessel Map

Figure 4(a) shows the result of the 2-D Frangi filter²⁷ on the single-band cross-polarized 470-nm channel, which identifies the vasculature structure using a colormap. The values range from 0 to 1, where larger values correspond to a more confident identification of a blood vessel. Figure 4(b) overlays the result in red on the input image for visualization and comparison purposes. The blood vessel map in Fig. 4(c) is extracted from the vasculature structure by negating and Gaussian filter smoothing. The dark areas identify blood vessels that should be avoided to prevent stricture. Blood vessel avoidance is achieved by elementwise multiplication of the blood vessel map to other maps. Figure 4(d) visualizes the blood vessel map on the input 470-nm band image.

Fig. 4

(a) Blood vessel segmentation using Frangi 2-D filter.²⁴ (b) Blood vessel segmentation result (red) image overlay on the single-band image at 470 nm. (c) Blood vessel map $V(u,v)$ created by Gaussian filter smoothing of the output of the Frangi 2-D filter. (d) Image overlay of inverted vessel map (inverted for better visualization) on the single-band reflectance image of the intestine.

3.3.

Thickness Map

Thickness differentiation was performed using SAM.^28,29 A pilot study for thickness measurement, as demonstrated in Fig. 5, involved the use of three controlled bovine colon samples with layer heights of 0.75 mm (S1), 7.27 mm (S2), and 9.72 mm (S3). The mesenteries attached to the intestine were considered as separated tissues, which were extracted before the thickness analysis.

Fig. 5

(a) Digital photograph of three bovine colon tissues with different specified thicknesses (units in mm). (b) Thickness differentiation using the SAM method, with a thickness-corresponding colormap. The mesentery is indicated as a different tissue in green color. Black arrows: thicker tissue areas; white scale bar: 10 mm; and colormap unit: mm.

Figure 5(b) shows the analyzed tissue thickness indicated by a heat colormap to represent thickness ranging from 0 to 10 mm. The result was mostly consistent with the measured physical dimensions of each sample, as shown in Fig. 5(a). In addition, it also indicates the effect of tissue types on thickness analysis, especially in the situation of a thin nonhomogenous sample such as S1, where blood vessel at similar height as S2 is classified as having the same thickness as the sample S3 (Fig. 5, black arrows).

Similarly, we applied the same SAM method to the multispectral images of the same porcine intestinal tissue in order to extract thick tissue area, as it is highly influenced on applied suture tension and bite size. Bowel wall thickness increases with age from 0.5 mm for infants to 2.0 mm for adults based on ultrasounds³¹ and is generally thicker than 0.9 mm in healthy pediatric and adult subjects.³² Thus, we are considering a lower limit of wall thickness for suturing to be 1 mm. We prepared our bowel tissue sample with a scalpel to contain a thinned-out section with thickness $<1\mathrm{mm}$ on the left (Fig. 6, orange, corresponding to a 0 for the suture map calculations) and a thicker section of about 2 mm thickness on the right (Fig. 6, red, corresponding to a 1 for the suture map calculations). Figure 6 shows the tissue thickness analysis using the supervised SAM method with predefined endmember references of thicker tissue section at the double-layered intestinal region (in red color) and thinner tissue section at the single-layered intestinal section (in orange color). The single-layered region indicates the incised section, while the double-layered one indicates a nonincised region. Prior to the SAM analysis, the image is analyzed to segment features such as the mesentery (in green) or blood vessels (in yellow).

Fig. 6

(a) Representative single-band reflectance image at 470 nm. (b) Thickness differentiation using the SAM method, with a thickness-corresponding colormap. The red color shows the thicker layer, and the orange color shows the thinner layer. Tissue classification of the mesentery (green) and blood vessels (yellow) was performed prior to the thickness analysis. White scale bar: 2 mm.

We further implemented a smooth Gaussian kernel convolution on this nonincised region. The smoothing kernel is used on the thickness binary map to signify the necessary thick tissue density for the convolution of $B(u,v)$ and $V(u,v)$ maps (Fig. 7).

Fig. 7

(a) Thickness binary map as evaluated by the SAM algorithm. (b) Smoothed thickness map $T(u,v)$. Larger values (brighter) denote areas with thicker tissue, which are better suited for suture placement. (c) Overlay of thickness map over a single-band image for better visualization.

3.4.

Multispectral Tissue Classification and Bite Depth Map

The multispectral tissue analysis results using the MultiSpec program are shown in Fig. 8. The composite image [Fig. 8(a)] of the porcine intestinal tissue was created from three spectral band images of the background, lumen (mucosa and submucosa layers), blood vessels, thick tissue outer layer of serosa, and the mesentery. Those four regions were segmented in different colors [Fig. 8(b)]. The foreground mask is shown in Fig. 8(c). The lumen, blood vessel, and thin tissue areas of serosa and mesentery are shown in red [Fig. 8(d)] and yellow [Fig. 8(e)], respectively, whereas the mesentery is depicted in green [Fig. 8(f)]. Although a small tissue portion including the blood vessels was indicated in red, the program successfully segmented the inside and outside tissue areas and the mesentery. The blood vessels are accurately accounted for in the final map, using the specific blood vessel segmentation results (Sec. 3.2). We can observe that lumen, blood vessel, thin and thick tissue regions of the intestine, and mesentery are successfully segmented by the algorithm after repetition of training sets for several tissue types as supervised by the user. The inside (mucosa and submucosa) and outside (serosa) of the lumen can be used to extract the edge between the two regions, which outlines the cut line [Figs. 9(a)–9(c)]. The cut line is used as a reference to place sutures at a certain distance for better healing. To extract the edge regions, we used the standard binary image processing methods of dilation and erosion in MATLAB (R2015a, Mathworks Inc., Massachusetts). This line [Fig. 9(c)] was used to determine the bite size distance, given the tissue type and size. In surgery, the rule of thumb for the general suture technique is to calculate the suture placement distance from the tissue cut end (suture width) as 1.5 times the tissue thickness, $\tau$.²³ The computed bite size distance was convolved with a smooth Gaussian to account for uncertainty in the bite size distance computation. The standard deviation $\sigma$ of the Gaussian filter is chosen to give more weight to points that are at $1.5\tau$ (mm) distance but are not closer than $\delta$ mm from the cut edge. For 99.7% of the filtered values to be in this range ($3\sigma$-rule), $\sigma =(1.5\tau -\delta )/3$. The value of $\delta$ depends on the suture size but usually should not be smaller than 0.5 mm. With a tissue thickness of 1 mm for this sample, $\sigma =0.33\mathrm{mm}$. The resulting map, $B(u,v)$, is depicted in Figs. 9(d) and 9(e).

Fig. 8

Multispectral tissue classification: (a) composite image created from three spectral band images; (b) classified image using the supervised classification algorithm; (c) background color-matching image (black); (d) image showing the vulnerable tissue (lumen, blood vessels, and thin tissue regions) (red); (e) image showing the thick tissue regions (yellow); and (f) image showing the mesentery (green).

Fig. 9

Multispectral image analysis facilitates segmentation of (a) outer layer of the lumen (serosa) and (b) inner layers of the lumen (mucosa and submucosa layers). (c) The cut edge is automatically extracted by pixelwise multiplication of a dilated map of outer and inner layers of the lumen. (d) A bite depth map $B(u,v)$ is generated by translating and smoothing the cut edge by 1.5 times the thickness of the tissue. (e) Overlay of the bite depth map on a single-band image for better visualization.

3.5.

Suture Map and Suture Placement Recommendations

Finally, a suture map including the anatomical and geometrical information was generated using the smooth gradients from the individual image analyses (Fig. 10). The bite depth, tissue thickness, and blood vessel maps [Figs. 10(a)–10(c)] are combined using a multiplier operator to create the suture map, as shown in Fig. 10(d). Suture placement recommendations are the local maxima from the suture map and are depicted in Fig. 10(e) with the overlay image shown in Fig. 10(f). At the end, the surgeon is provided both the recommendations as well as a colormap overlay of the acceptable suture locations on the image, so that they can decide if they want to overrule the recommendation of the software (Fig. 11).