During surgery, a surgeon has to remove pathological lesions, replace abnormal tissue, or reconnect or repair damaged tissue structures, all of this while healthy tissue areas have to be maintained and organs at risk have to stay completely untreated and healthy. In order to accomplish this objective, to remove pathological while saving healthy tissue, the surgeon has to differentiate between healthy tissue areas and abnormal or damaged tissue. This manual differentiation process is very complex as the visual occurrence of living tissue, whether healthy or abnormal, often shows no significant optical differences under white light illumination. Furthermore, pathological structures mainly develop between or in the direct proximity to important structures and have most fine extensions. These facts make it hard to distinguish pathological from healthy tissue and, as a consequence, the structure exposition process is of high risk and a significant degree of difficulty, because damage to healthy structures can cause a temporary or permanent loss of functionality of the affected region.
Nowadays, the continuous differentiation during surgery is only based on the surgeon’s perception, experience, and anatomical knowledge, which entails great risk because abnormal tissue in the operating field displaces normal structures and changes the anatomy. Furthermore, since tissue structures are very hard to separate using optical and haptic qualities, this manual differentiation process must be done slowly with the utmost caution because injuring important structures, such as nerves, can cause permanent damage to the patient’s health.
During most surgeries, external tools are used to support the surgeon’s decisions on tissue differentiation. The intraoperative neurophysiological monitoring, e.g., electromyographic monitoring of peripheral nerves, is an external electrophysiological method to monitor the functional integrity of certain neuronal structures.1 To use this tool, the surgeon has to interrupt dissection and needs to change the instruments, which takes time and disrupts affiliated actions.
The upcoming digitization of information during surgery provides new opportunities of analyzing and visualizing tissue. In order to support the surgeon’s decision by detecting optical tissue characteristics not visible for human eye, we developed and investigated a hyperspectral method for analysis of in vivo tissue characteristics of different tissue types. Hyperspectral imaging (HSI) combines spectroscopy and imaging techniques to acquire spectral and spatial information of a scene.
The aim of this project is to perform first pretrials in noninvasive tissue differentiation using continuous illumination in the visible and near-infrared (IR) spectral range without using any fluorescent dyes. This approach has a number of benefits: there is no risk of side effects or allergies to a fluorescent dye and the treatment can be started without any delay caused by the enhancement of the tracer. Suitable monochromatic light allows one to stimulate endogenous and exogenous dye molecules.2 Energy is emitted in the form of light-specific wavelength, resulting in tissue-specific absorption, reflection, and scattering. Attention must be paid to the optical window of human tissue for near-IR illumination, as the absorption of light is minimal in the spectrum of 660 to 1100 nm.3
Tissue consists of various molecular structures, which imply different electromagnetic interactions and, therefore, different optical characteristics in terms of reflection, transmission, and absorption for different wavelength of the individual tissue types.4 There have been several attempts to examine optical properties of human tissue, mainly focusing on types that can be clearly separated from surrounding tissue, e.g., brain tissue, lung, and skin.2,45.6.–7 Other works are focused on tissue differences caused by different blood supply characteristics.2 In contrast, the present investigation analyzes the optical properties of human tissue that suffer of large errors. Mostly, the characteristics of specific tissue types are analyzed in vitro.8,9 Due to the degeneration of human tissue without blood supply as well as changed water and collagen content, the optical properties are assumed to change between in vivo and in vitro analysis.7 Therefore, the tissue behavior will be analyzed in vivo in this work.
HSI is established in biomedicine and food science for cell segmentation, skin and meat analysis,2,10,11 as well as for recording the properties of blood by the detection of differences of oxygen saturation of hemoglobin. However, it has not yet been used in intraoperative medical therapy for image-guided tissue differentiation. Different techniques exist to acquire the hyperspectral data of a scene. To study molecular chromophors as oxyhemoglobin in a patient, liquid crystal tunable filters and digital light processing have been used.12,13 Furthermore, hyperspectral cameras can be used with linescan or snapshot mode to acquire the dataset in a short amount of time, although with low resolution.14,15 Alternatively, a filter wheel in front of the sensor or behind the light source can step through the hyperspectral space.16,17 Since the method using the filter wheel allows higher resolution images, and this work does not aim at real-time results, we decided to use a filter-wheel setup.
The different optical tissue characteristics can be traced back mostly to the different water and collagen content as well as to the fiber development of the tissues.2 Water strongly absorbs IR and ultraviolet (UV) light and shows no significant absorption for the visible electromagnetic spectrum.18 In the spectra of near-IR, visible and near-UV melanin and hemoglobin show high absorption. The differences of the optical behavior of hemoglobin (in the veins) and oxyhemoglobin (in the arteries) are well-known and used in narrow-band-imaging (NBI).19 Hemoglobin and oxyhemoglobin have the largest difference in absorption at about 415, 560, as well as 660 nm. NBI uses this knowledge to differentiate between arteries and veins.
In this paper, we present a hyperspectral filter-wheel setup, which is used to analyze optical human tissue properties in vivo. The possibility to sequentially distinguish between arteries and veins is examined in the context of this work to verify our method. Furthermore, other tissue types are analyzed to examine whether it is possible to differentiate between different tissue structures. In this study, only healthy tissue structures, well-known by the surgeon and existing in every analyzed patient, are analyzed to minimize uncertainties about different tissue behaviors. Therefore, the optical properties of fat, muscle, connective tissue, nerves, parotid gland, and bone are analyzed and presented.
Materials and Methods
We build an imaging setup based on a prototype of the digital surgical microscope ARRISCOPE including a digital camera ARRI Alexa (Arnold und Richter Cine Technik, Munich, Germany) featuring a CMOS sensor as recording device and an illumination unit. In front of the illumination unit, a filter wheel selects a specific wavelength band that reaches the tissue. The use of a filter wheel to achieve multispectral illumination is described by Brauers et al.16 This approach allows a spectral scanning of the investigated tissue. The complete setup is sketched in Fig. 1.
A microcontroller manages the whole illumination and acquisition process. The light source is a white LED, which emits light between 400 and 700 nm. The spectral profile is shown in Fig. 2. The luminous flux is 2400 lm.
When the acquisition process is started, the camera back end sends rectangular trigger pulses to the controller. The filter wheel is then synchronously rotated by the motor with the aid of a light barrier and an angle sensor. The filter wheel has a diameter of 21.8 cm and includes 16 bandpass filters with wavelengths starting at 400 up to 700 nm in steps of 20 nm. The bandwidth in terms of full-width half-maximum of every filter is about . The light is guided using a fluid light guide to achieve a higher illumination. The optimal light transmission has a spectrum from 340 to 800 nm. In the optical path in front of and behind the filter wheel, a convex lens is needed to widen and narrow the light beam. At the end, different monochromatic light leaves the optical system, and the spectral changes are synchronized to the camera frame rate.
The camera uses a CMOS sensor with a sensitivity range from 380 to 1100 nm, see Fig. 2. The best working spectrum is between 400 and 880 nm. The sensor and camera optics specifications are listed in Table 1. The optimal working distance (WD) to the surgical area is 210 mm, which corresponds to a normal surgical microscope.
The sensor and camera optics specifications in the used system.
|CMOS sensor number of pixels|
|Output resolution||Full HD|
|Diagonal dimension||29.26 mm (1.7 in.)|
|Camera optics specifications|
|Beam angle||Wide angle||16.52 deg|
|Focal length||Mean||58.237 mm|
|Wide angle||64.523 mm|
|Field of view at 210 mm WD||Max.|
We have investigated eight different tissue types, which are scanned in three different surgical procedures. The acquisition process has been a noninvasive approach by which the reflection of the different tissue types of the monochromatic illumination is captured. Only tissue that has been exposed within the normal surgical treatment has been scanned with our setup additionally to the normal surgical record and approved by Charité, Germany. The surgical procedures are mastoidectomy, parotidectomy, as well as neck dissection. We have scanned six patients and acquired datasets at different times during surgery. This results in several different acquired scenes with a view on different tissue types. The evaluated tissue types are artery, vein, bone, muscle, fat, connective tissue, parotid gland, and nerve.
While scanning the surgical area with our hyperspectral setup, it is important to avoid scattered and indirect light from the external sources. Therefore, the ceiling light has to be turned off and the operating light system must either be turned off or at least dimmed and shifted out of the surgical area. In first measurements without turning off the ceiling lights and dimming the operating light, too much distracting scattered light was visible in the data and no significant results could be achieved.
If the light conditions are satisfactory and the image requirements are met, the data recording can be started. During recording, the filter wheel rotates and provides continuous changing monochromatic illumination of the surgical scene. The camera acquires a monochromatic movie of the scene.
Using the measured wavelength information in every dataset, a three-dimensional (3-D) hyperspectral (spatial—, ; spectral— dimension) data cube is calculated for evaluation. Since the rotation of the filter wheel leads to a continuous vibration of the mechanical setup, a slight motion shift among the acquired monochromatic images is noticeable (external movement). This external movement in the spectral dimension has to be handled during the postprocessing steps.
We have analyzed the spectral tissue behavior in the wavelength domain. To analyze the complex information in the images, a few postprocessing steps are needed to minimize the existing movement and luminance intensity differences in the images. First, every acquired dataset is annotated by the surgeon to get a well-known tissue classification. Based on this annotation, every image can be segmented into subimages, each containing one relevant tissue type only (Fig. 3).
In addition to the external movement of the imaging system, the cardiac cycle as well as the ventilation lead to internal movement. To transform the measured data into the wavelength domain, the internal movement along the spatial dimensions and external movements along the spectral dimension in the hyperspectral data have to be minimized. Therefore, the images of all wavelengths are divided into subimages according to the annotations of the surgeon so that each subimage includes only one tissue type. The (sub-)images of the corresponding tissue in all wavelengths of a dataset are registered to achieve a 3-D hyperspectral data cube. This registration process of two (sub-)images and is done by analyzing local translation of each annotated region only since translation is the main factor of dislocation.20,21 For registration, we have used two different methods and picked the one with the smallest variation in the resulting 3-D dataset. The first method minimizes mutual information
As it is shown in Fig. 2, the intensity of the light source as well as the sensitivity of the sensor vary widely. To compensate this, the measured hyperspectral data are adapted to handle different illumination intensities for every spectral band using a normalization
To evaluate the accuracy of our hyperspectral analyzer, we have compared the spectral behavior of detected artery and vein data with well-known published oxygenated and deoxygenated blood results. Equal spectral behavior has been observed here.
In Fig. 4, 12 out of the 16 registered monochromatic images of a single scan are shown. A side-by-side comparison of these images already reveals significant differences in several parts of the image and shows at which wavelength the different vascular structures can be differentiated.
During the analysis, one big challenge is the huge variance in the intensity of the light source. Despite intensity normalization, it is still difficult to achieve proper results for all wavelengths because the illumination intensities at 400, 680, and 700 nm are near the limit of detection. Table 2 shows the signal-to-noise ratio (SNR)Fig. 4, where the image obtained at 400 nm determines noise as it contains almost no signal but noise.
The signal-to-noise behavior of the acquired images of Fig. 4, for each wavelength. The image at 400 nm contains almost no signal and is thus used as noise reference.
In order to achieve comparable results, we have adjusted the wavelength information according to its corresponding input illumination. For a better comparison of the different optical tissue behavior and presentation in this publication, we additionally normalized every spectrum using
First of all, it is of interest if the same tissue types show the same behavior over different measurement times for one patient as well as between different patients, which includes changes in illumination caused by indirect background lightning. As shown in Figs. 5 and 6 for bone, fat, parotid gland, and nerve, the behavior of the normalized reflection intensity for the analyzed tissue type remains the same for different measurements and individuals in the analyzed spectrum from 400 to 700 nm. The spectrum of bone [Fig. 5(a)] shows its highest intensity at 420 nm with three additional smaller peaks at 460, 520, and 580 nm.
The spectra of fat [Fig. 5(b)] and parotid gland [Fig. 6(a)] are completely different to bone. Both show highest intensity at 460 nm with a smaller second peak at 580 nm. The differences in these two spectra are the intensity level in the green (480 nm) to red (640 nm) spectral range. The spectrum of the parotid gland shows a small third peak at 520 nm and remains relatively constant in the range of 480 to 560 nm while the spectrum of fat is slightly descending in this range.
The analyzed reflection behavior of nerve tissue is more complex, see Fig. 6(b). At this spectrum, two high peaks at 460 and 580 nm as well as a smaller third peak around 540 nm exist. The variability in the spectrum of the nerve is higher than for the other spectra, which can be clearly seen at 420 and 520 nm.
Figures 5 and 6 clearly show that every single tissue type shows an individual unique trend which allows a precise identification whereas the standard optical appearance allows mostly no accurate identification for the surgeon. During the annotation of the data, the surgeons were not completely sure about the type of tissue for certain areas. Based on their knowledge, the possible tissue type could be narrowed, but not clearly defined. Figures 7(a) and 7(b) show the two well-defined curves of muscle and connective tissue, respectively. Figure 7(c) shows the behavior of tissue that is classified by the surgeons as either muscle, connective tissue, or a mix of both. The curve of patient 4, first measurement (P4/1) can be definitely identified as muscle. The curve of patient 2, first measurement (P2/1) appears to be a mix of both types, but the tissue part contains more muscle than connective tissue. Therefore, the spectrum is more similar to the spectrum of muscle.
The spectrum of connective tissue also reveals a large variability, which can correspond to the different types of connective tissue. This requires further analysis.
Figure 8 combines all unique spectra of the analyzed tissue types. While bone and connective tissue show a high reflection in blue at 420 nm, they differ very much in the range of 500 to 580 nm. In the range of 500 to 580 nm, fat shows a similar trend to bone, but the blue content is similar to muscle or parotid gland. In the green to red range (520 to 580 nm), all other tissues exhibit a higher reflection than bone and fat with high variability and very unique characteristics. Therefore, up to seven spectral points can be identified to differentiate all analyzed tissue types. The higher the difference between two tissue types at these spectral points, the better the different types can be separated.
We have analyzed the spectral tissue behavior in the wavelength domain. It is shown that our setup is robust and allows us to state reliable conclusions about tissue behavior. As a result, we can present a spectrum that could allow a characterization of the tissues in the wavelength domain and find specific wavelengths to separate the different tissue types in vivo, e.g., bone, muscle, or fat. Interesting wavelengths are in the ranges of 420 to 460 nm and 530 to 590 nm as well as 640 to 680 nm. As indicated in Fig. 8, tissue differences can be identified at the spectral points 420, 440, 520, 540, and 640 nm. In addition, for some tissue types, near-UV and near-IR illumination could bring more significant differentiation options, which need further investigations, e.g., bone in near-UV.
Using these interesting wavelengths and projecting the different measured tissue data, single tissue types can be discriminated from the other tissue types as shown in Fig. 9 for the parotid gland. These fundamental investigations give very promising results and can help to assist or even simplify certain surgical procedures.
The main problem in our setup is caused by inhomogeneous illumination in the spectral range. Using a diffuse light source with a wider and homogeneous spectral range as a xenon source could reduce the observed distortion as well as noise parameters and is currently under investigation. On the other side, the used light source should not be too strong in terms of intensity as otherwise distracting reflections at vascular walls and fluids reduce the information output.
Another large source of deviation and error indicator is the existing scattered light in the operating room caused by normal ceiling and surgical lights. Our study showed that the best results could be achieved without any scattering light, which is not possible to put into practice because the physicians and surgical nurses need lighting as well. Therefore, a reasonable compromise between room illumination and optimal darkening has to be found.
To take the approach presented here closer to a practical application, the setup has to be optimized addressing the following points mainly: internal and external movements (pulse beat and breathing as well as camera vibration, in and between the images, respectively) need to be minimized to avoid expensive image postprocessing and speed up the image analysis. In addition, the existing time misalignment, caused by the cycling through the different bandpass filters, which is too slow and therefore not applicable for clinical routine, has to be prevented in an optimized setup. Caused by these facts, setup changes will be investigated in future. Further, the position of the wavelength filters will be analyzed in more detail since there are different possibilities of placing the filter in the setup. In this work, the filters are placed directly behind the light source, but it is also possible to place them in front of the sensor (hyperspectral cameras). Further, a possible approach is to combine both possibilities: an illumination unit consisting of different filters with a wider spectrum and a sensor also with filters in front, e.g., an RGB sensor.
We developed and investigated a hyperspectral analyzer to detect optical tissue characteristics not visible for human eyes. These characteristics have the potential to assist the surgical procedure by providing additional information about the visible tissues to the surgeon. This would induce a faster treatment and has the potential to reduce the risks of postoperative damage.
In future work, we will optimize the setup as described and plan to adapt the light source to achieve homogeneous illumination as well as to extend the spectral range to near-UV and near-IR in order to potentially detect further interesting tissue behavior and thereby make a potential tissue differentiation more robust. Further, it is planned to use a snapshot hyperspectral camera despite the disadvantage of the lower spatial resolution and analyze the tissue behavior using white light illumination to check if the results are comparable to the ones presented here. Following, an optimal setup will be developed using information convolution. Thus, several narrow-band signals can be achieved in a shorter time and fewer images, which would balance the setup in terms of spatial resolution and acquisition time for real-time capability. Using an optimized setup that acquires the data in less time allows to optimize the postprocessing, mainly the registration process can be extended to more comprehensive models, e.g., as presented by Furch et al.22 In addition, we will investigate other interesting tissue structures, healthy as much as diseased tissue, in further surgical procedures.
Once the tissue differentiation is robust, the visualization options will be discussed and analyzed. The visualization should occur directly in sight of the operator without distorting the environmental truth. Therefore, our idea has the potential to optimize surgery time and costs.
The authors have no relevant financial interests in this article and no potential conflicts of interest to disclose.
The authors like to thank Arnold und Richter Cine Technik, Munich, Germany, for providing a camera and the prototype of an optical microscope.
Eric Larry Wisotzky received his BSc and MSc degrees from Heidelberg University, Germany, in 2012 and 2014, respectively. After a research visit at Sydney Medical School at the Sydney University, he is currently working toward the doctoral degree at Humboldt University of Berlin and Fraunhofer Heinrich-Hertz-Institute. Together with the Department of Otolaryngology of Charité Berlin, he is working on new image-guided surgery techniques. His research interests span from imaging physics to image processing and modeling in medicine.
Peter Eisert is professor for visual computing at the Humboldt University of Berlin and heading the Vision and Imaging Technologies Department of the Fraunhofer HHI, Germany. He received his Dr.-Ing. degree in 2000 from the University Erlangen and worked as postdoctoral fellow at Stanford University. He is coordinating and initiating numerous research projects on image/video analysis and synthesis, face processing, computer vision and graphics in multimedia, security, and medical applications. He has published more than 150 conference and journal papers and is an associate editor of JIVP and JVCI.