2.1.

Experimental Setup

A standard clinical otoscope was used as the base optical setup for our real-time 3-D scanning device. In its standard configuration, the otoscope consists of a handle and a head with at its distal end an attachment for disposable plastic specula. Uniform white light is generated in an LED module located inside the otoscope handle and is injected into the ear canal via a plastic mirror. The light that is reflected from the TM is focused onto the observer eye by a $3\times$ magnifying lens placed at the rear of the head.

To provide the otoscopic device with digital fringe projection and 3-D data extraction capabilities, several modifications to its optical configuration were implemented. First, the LED module and the entire otoscope handle were removed from the otoscope head. The plastic mirror was replaced by a semitransparent mirror (SM) that was installed inside the otoscope head at a 40 deg to 45 deg angle between central speculum axis and former handle shaft. Next, a custom-handheld aluminum frame was designed to align a high-speed CCD sensor (Imperx IGV-B0620M, 210 frames per second, $640\times 480\text{pixels}$, 12-bit grayscale images) perpendicularly to the prior otoscope handle axis. Finally, the magnifying lens was removed and a Texas Instruments (TI) lightcrafter pico projector evaluation module was installed at the rear of the otoscope head. The modified otoscopic unit and a schematic overview of its optical layout are shown in Fig. 1. In its entirety, the prototype weighs around 800 g and can easily be held in one hand. The device is powered and connected to the PC only through flexible cables and can easily be maneuvered under any angle. Nevertheless, future iterations of the hardware design will include more detail to ergonomics and form factor.

Fig. 1

(a) Modified otoscopic unit and (b) schematic overview of its optical layout with enlarged detail of otoscope head.

Uniform red, green, or blue light is generated by the RGB-LED light engine inside the TI lightcrafter module and reaches the DMD under a fixed angle. The light is reflected by the individually rotating micromirrors into a pixelated ($608\times 684\text{pixel}$) grayscale fringe pattern that is projected onto the object surface after passing the projector lens and the SM. The deformed fringe pattern is observed by the CCD sensor after being reflected on the SM and after passing the observation lens. The coaxial bayonet neill–concelman cable enables triggering between CCD and DMD, and the recorded fringe patterns are transferred from CCD memory to RAM memory using the gigabit ethernet interface.

The TI lightcrafter module is a compact ($116\mathrm{mm}\times 65\mathrm{mm}$) DLP-based light projection system containing flash memory for on-board pattern and sequence storage. It generates 8-bit grayscale patterns at a rate of 120 Hz and binary patterns at a rate of 4 kHz. The pico projector and its focusing lens system was oriented so that the central projection axis of the projected images enters the head under an angle of $\alpha =20\mathrm{deg}$ relative to the observation axis. This angle between projection and observation axes causes the projected fringe patterns to appear deformed by the object shape and allows the 3-D object surface data to be extracted.

The duty cycle of the rapidly rotating micromirrors determines the amount of light that is reflected toward the output and therefore sets the grayscale intensity per pixel. In contrast to commercial video projectors, which adopt nonlinear duty cycles for better subjective visual effect, the lightcrafter module is designed to maintain a linear relation between input and output grayscale intensity or “gamma curve.” The micromirror duty cycle is divided discretely into 8-bit planes, each representing a fraction between 1/2th and 1/256th of the total illumination time. Therefore, to avoid visual artifacts, it is important that the projector-camera system is synchronized to record integer multiples of the total illumination time only. To ensure proper triggering, custom-made electronics were built to connect the trigger output of the lightcrafter logic board to the high-speed camera, and the camera exposure time was set to the interval between successive trigger pulses, corresponding to 1/120th of a second or 8.33 ms per image capture.

2.2.

Real-Time Digital Signal Processing Pipeline

The digital nature of the DLP unit allows us to implement any structured light projection technique in the otoscopic profilometer. Without loss of generality, in the following we describe the implementation of the standard three-phase $2+1$ technique.²⁶ This technique requires only three unique intensity images per 3-D measurement and therefore allows a high 3-D frame rate to be obtained. For a similar description of four-phase fringe pattern pipelines, we direct the reader to Van der Jeught et al.⁵

Before acquisition, three preconstructed 8-bit digital patterns are uploaded to the on-board flash memory of the lightcrafter module and are displayed in a loop: two line patterns with sinusoidally varying intensity distribution and a relative phase shift of $\pi /2$ and a single uniform white image. When observed by the camera under a relative angle with the projection axis, these line patterns appear deformed by the object shape and can be described as follows:

To achieve real-time display of 3-D height maps, several optimization techniques were employed in the proposed setup. First, acquisition, digital signal processing, and visualization of height maps are executed simultaneously in an asynchronous workload organization. Second, all digital signal processing components themselves are designed to operate in parallel on the graphics processing unit (GPU) of a standard graphics card. Third, GPU interoperability between parallel data processing interface compute unified device architecture (CUDA) and visualization interfaces, OpenGL and GLSL, were established to minimize data transfers across the PCI bus and to relieve the CPU from most computationally intensive tasks. The entire software library, from data acquisition to onscreen rendering of 3-D measurement results, was written in custom C++-based code to ensure optimal data flow control and memory management. The digital signal processing pipeline was executed on a commercial GTX980 graphics card with 3072 CUDA cores and 4096MB of on-board GDDR5 RAM memory.

A more detailed schematic overview of the real-time $2+1$-phase profilometry pipeline is presented in Fig. 2. Input fringe patterns $I_{1-3}$ are acquired continuously and are downloaded from the camera data buffer in blocks of three. This process is supported by hardware triggering and controlled by the CPU. Simultaneously, the previous block of fringe patterns is transferred from host (CPU) to device (GPU) memory, where the 12-bit grayscale input images are stored as 2-byte unsigned integer texture arrays. Next, control over the data processing pipeline is passed over to the GPU and the CPU returns to data acquisition control. The first step in the digital signal processing pipeline is the extraction of the wrapped phase ${\varphi }_{\mathrm{W}}$ from the three input fringe patterns using Eq. (4). As the phase of neighboring pixels can be calculated independently, this process is highly suited for parallel implementation. Next, the original, continuous phase signal ${\varphi }_U$ is recovered by adding an integer multiple of $2\pi$ to every phase value of the wrapped phase grid ${\varphi }_W$ in a process commonly referred to as 2-D phase unwrapping.²⁷ A $3\times 3$ Gaussian filter kernel is applied to the unwrapped phase array to reduce spikes caused by motion or system noise, and the filtered phase values ${\varphi }_f$ are converted to actual 3-D Euclidian coordinates by applying the reference-plane-based calibration approach.²⁸ After recording the phase grid of a reference plane ${\varphi }_R$, where the base plane depth is set to $z=0$, the difference between ${\varphi }_R$ and subsequent phase measurements is proportional to the object depth $z$ by some constant scaling value. This scaling value $z_S$ can then easily be obtained from calibration measurements on an object of known dimensions. Finally, point normals to the vertex geometry $z_N$ are calculated by taking the cross products of neighboring phase value gradients so that Phong²⁹ lighting may be applied in 3-D visualization mode. Note that the digital signal processing blocks as shown in Fig. 2 are not scaled to represent the respective time they take but are in fact expanded to fill the maximum time slot they could take up without obstructing the real-time pipeline. In reality, the entire digital signal processing of the three images, including memory transfer from host to device, takes up only 11.2 ms in total.

Fig. 2

(Color online) Real-time three-step $2+1$-phase shifting profilometry timeline. Horizontally aligned blocks occur sequentially; vertically aligned blocks occur simultaneously. At all times, fringe pattern acquisition, digital signal processing, and on-screen rendering of successive measurements are executed asynchronously. Following the data-processing cycle of one 3-D-measurement (blue, diagonally hatched), three input fringe patterns are first captured by the high-speed camera and transferred from CPU to GPU memory. Next, the digital signal processing pipeline performs successive wrapped phase extraction (${\varphi }_W$), phase unwrapping (${\varphi }_U$), Gaussian filtering (${\varphi }_f$), phase-to-height conversion and scaling ($z_S$), and normal calculation ($z_N$). Finally, the 3-D height map is rendered and displayed.

The GPU-calculated height measurements are visualized from GPU memory and no additional device-to-host memory transfers are needed. For final rendering of the result, two graphics application programming interfaces are employed: OpenGL and GLSL. To allow off-screen rendering, i.e., the drawing of scenes into buffers other than the frame or screen buffer, a standard double buffering technique is employed. To this end, an additional OpenGL frame buffer object is created as a backbuffer and its output is bound to GPU texture memory. Only when the backbuffer is fully rendered, it is swapped with the screen buffer or front buffer, effectively hiding the rendering process from the user while minimizing latency. For final display, three small shader programs are written in the OpenGL shading language GLSL. The first one incorporates the previously calculated point normals in a 3-D Phong light reflection model, computing the virtual illumination brightness and color of the object surface pixel per pixel, depending on user-defined variables such as object surface reflectance and 3-D position of light source and observer. The second one converts the height values to a pixel color using a predefined color map, generating a 2-D color-coded height map that is projected onto the object surface for quick assessment of object topography. The third one projects the uniform bright image that was gathered in the acquisition step of the $2+1$-phase technique onto the object surface. This shader program enhances operator orientation as object texture features that cannot be resolved as height differences are incorporated into the final 3-D result, also.

3.1.

Measurement Precision

To determine the depth resolution of the proposed otoscopic profilometry setup, the surface profile of a flat glass plate was measured at various discrete distance intervals around the central focal plane. Pregenerated structured light patterns with a sinusoidal intensity profile of 24 pixels per fringe period were projected onto the glass plate surface. The glass plate itself was mounted onto a translation stage, which allowed displacement along the projection (or $Z$-) axis. The projected fringe patterns spanned a field of view of $8.15\times 6.11\mathrm{mm}$ across the glass plate surface, which was coated with magnesium oxide to ensure good diffuse reflectivity. Cross sections of these measurements along the central pixel lines in both $X$- and $Y$-dimensions are shown in the top row of Fig. 3. The numerical difference between these cross sections at depths of $Z=+4\mathrm{mm}$ (red), $Z=0\mathrm{mm}$ (black), and $Z=-4\mathrm{mm}$ (blue) around the central focal plane are included in the second row. No Gaussian smoothing was applied to the 3-D data.

Fig. 3

The surface profile of a flat glass plate was measured at discrete intervals around the central focal plane $Z=0$ to determine the measurement precision of the proposed otoscopic system. Cross sections of these depth measurements along (a) the $X$-axis and (b) the $Y$-axis are included in the first row, and the corresponding depth measurement errors ($dZ$) at object depths of $Z=+4\mathrm{mm}$ (red), $Z=0\mathrm{mm}$ (black), and $Z=-4\mathrm{mm}$ (blue) are included in the second row. (c) Plots the 2-D-RMS values of the difference between the full flat glass plate measurements and their respective best planar fits in function of object depth.

It can be seen that measurement precision decreases with increasing distance of the object to the focal plane. This effect is standard for structured light projection techniques due to the fact that the contrast of the projected fringe patterns deteriorates away from the focal plane due to defocusing effects. With decreasing fringe pattern contrast, the intensity values $I_{1-3}(i,j)$ in Eq. (4) converge toward each other. This reduces the numerical digital range of the intensity ratio formula used to calculate the phase $\varphi (i,j)$ and limits the measurement resolution of the technique. Furthermore, it should be noted that the projection optics of the TI lightcrafter module are designed for asymmetric upward projection. When the projected fringe patterns reach the object surface outside of the focal plane, they contain a certain amount of geometric distortion relative to the angle between the normal of the object surface and the central projection axis. The accumulated measurement errors due to fringe pattern defocus and geometric distortion artifacts largely remain within a region of $dZ=\pm 200\mu \mathrm{m}$.

To quantize the system’s mean measurement error as a function of object depth, the full-field two-dimensional root-mean-square (2D-RMS) values of the differences between the flat plate measurements at each depth interval and their respective best 2-D plane fits are plotted in part of Fig. 3(c). From this plot, axial resolution of 3-D measurements can be determined, given the total object depth span and assuming that the object is placed centrally around the focal plane. The measurement error in depth ($dZ$) increases with distance away from the central focal plane, though remains below an RMS of $60\mu \mathrm{m}$ within a region of $\pm 4\mathrm{mm}$ around $Z=0$.

3.2.

Calibration Standard and Live Visualization Software

To demonstrate the real-time digital signal processing and visualization engine of the proposed otoscopic profilometry setup, a calibration standard with right-angled triangular ridges of 2.2-mm deep was measured. Again, the object was coated with magnesium oxide to ensure maximum diffuse reflection and its height profile was distributed symmetrically around the focal plane. A still frame from the live-recorded measurement video (Video 1) is shown in part of Fig. 4(a). The central pixel line of the resulting 3-D measurement is highlighted in red. The cross section of the height measurement along this central pixel line is illustrated in blue in Fig. 4(b) with its segmented linear fit overlaid in red. The numerical difference between height measurement and linear fit is included in Fig. 4(c).

Fig. 4

(a) Video 1 still frame from live measurement video demonstrating the real-time otoscopic profilometry technique, including a cross section of the object’s height profile $Z$ along (b) the central pixel line, and (c) its measurement error $dZ$. (Video 1, 5.65 MB, mp4 [URL: http://dx.doi.org/10.1117/1.JBO.22.1.016008.1]).

As can be seen in the measurement video, the software allows the user to rotate the object in 3-D perspective view and to control visualization effects, such as colormap range, position of virtual light source relative to the observer, and quality threshold of rendered height map data. The video stream was recorded directly from the computer screen using frame grabber software and was resampled from 40 fps to 29.97 fps in accordance with standard H.264 codec protocol. During live measurement, the user is able to switch between custom shader programs designed to emphasize either the height profile or the texture of the object surface. For fast assessment of the object’s height profile, the object vertices are each assigned a color ranging from blue (low $Z$-value) to red (high $Z$-value). Alternatively, the shader can superimpose the uniformly lit input intensity image onto the object surface to incorporate texture features in the 3-D model. Finally, a blend or a combination of these two shader programs can be selected, which is shown in the first part of Video 1 and Fig. 4(a).

To reduce high-frequency phase noise during live measurements, a $3\times 3$ Gaussian filter is applied to the unwrapped 3-D data. As a result, the 3-D geometry is smoothed out, and sharp corners become rounded. This effect can be seen clearly in Figs. 4(b) and 4(c) when measuring the right-angled indentations of the calibration standard. The measurement noise stays well within $dZ=\pm 50\mu \mathrm{m}$, except at the borders between successive segmented lines, where the height measurements deviate up to $dZ=\pm 100\mu \mathrm{m}$ from the linear fit. Since the otoscopic profilometer was designed mainly to measure the surface profile of human TMs that typically possess a smooth 3-D tent-like shape and generally do not contain features with edged transitions, rounding artifacts of sharp angles do not impede the intended 3-D TM shape measurements. Nevertheless, the Gaussian filter is optional and can easily be omitted from the digital signal processing pipeline by the user during live measurement.

3.3.

Dynamic Pressurization of Tympanic Membrane Ex Vivo

As a first step toward implementation of real-time tympano-topography in the clinical ENT office, the otoscopic profilometer was used to measure the 3-D shape distortion of an ex vivo TM caused by dynamic pressurization. A fresh (frozen) temporal bone was obtained from the Life Legacy Foundation, and the TM remained in situ during the measurements. To mimic an in vivo measurement setup, the ear canal was left intact, and the TM was optically accessible only by inserting a plastic speculum into the ear canal. A small drop of water base cosmetic dye was used to coat the TM with a diffusely reflecting white layer. The middle ear cavity was (de-)pressurized by inserting a small tube through one of the mastoid cells and by sealing the remainder of the mastoid cells airtight using two-component silicon paste. The air tube was connected to a pressure generator that was controlled manually. The measurement setup is shown in Fig. 5.

Fig. 5

Dynamic tympano-topography measurement setup. The otoscopic profilometer head contains a disposable speculum, which is inserted into the ear canal of an airtight sealed temporal bone. The middle ear cavity is pressurized by connecting a small tube through one of the mastoid cells to a pressure generator (not shown here).

Dynamic pressurization of the TM was established by alternating the middle ear cavity pressure between $-0.4\mathrm{kPa}$ and $+0.4\mathrm{kPa}$ several times over the course of a 7.1-s measurement period. The $2+1$ phase-shifting profilometry technique was used to monitor the 3-D deformation of the TM shape in real time, and the measurement results were downloaded to CPU RAM memory buffers and stored in computer memory for postprocessing and analysis purposes. Figure 6 shows the various steps in the reconstruction process as it is applied to one of the 284 sets of three input intensity images [Figs. 6(a) to 6(c)]. Two of these images contain sinusoidally varying intensity profiles with a relative phase shift of $\pi /2$ and the third one has a uniform intensity profile. Together, they can be combined to form a phase map of the object [Fig. 6(d)] from which the height profile is constructed after removal of the $2\pi$-discontinuities [Fig. 6(e)]. Next, the phase values are converted to absolute height values, Gaussian filtering is applied to the 3-D geometry, and a reference map is deducted from the resulting height map. As we are interested in monitoring the 3-D shape deformation of the membrane when pressure is applied over it, we have chosen to assign the first unwrapped phase map of the measurement stream (at pressure 0 kPa) as the default reference geometry to be subtracted from the subsequent phase maps. The resulting height deformation maps [Fig. 6(e)] are therefore to be interpreted relative to zero pressurization.

Fig. 6

Various steps of the $2+1$ phase-shifting profilometry pipeline during measurement of TM (left ear, medial view) shape deformation. (a)–(c) Input intensity images, (d) wrapped phase map, (e) unwrapped phase map, and (d) height deformation map.

It should be noted that a quality map is constructed in real time for every trio of input intensity patterns by calculating the intensity modulation [the $I^{\prime \prime }$-term in Eqs. (1) and (2)] pixel per pixel and by performing binary thresholding accordingly. This way, parts of the image that are out of focus, contain shadows, or are under- or overexposed can be excluded from the phase map and do not affect the resulting 3-D measurement. However, the user is able to set outer limits to the region of interest by marking the region with an interactive polygon, should manual intervention be required. The quality map threshold value and the boundaries of the binary mask can be modified in real time during acquisition. Incorporation of the dynamic quality map in the measurement pipeline increases the robustness of the technique during live measurement when local structural irregularities (e.g., the tissue of the ear canal partially blocking the view of the TM in the upper right side of the measurement sequence in Fig. 6) might otherwise induce measurement artifacts in the reconstruction of the 3-D phase map.

As shown in Fig. 7, the pressure change in the middle ear cavity results in height deformation of the TM. Part (I) shows the height deformation map of the TM when the applied pressure was near maximum at $+0.4\mathrm{kPa}$ overpressure in the middle ear cavity. The manubrium border of the malleus is marked by a dashed black line, and three distinct locations on the membrane surface along the manubrium (black), near the center of the posterior part of the pars tensa (blue), and near the side of the pars tensa toward the annulus (red) are indicated by asterisks. The corresponding height deformation profiles of the membrane at these locations are plotted in function of time in part (II) and reflect the pressure change cycles clearly. It can be seen that the TM surface along the manubrium (black line) hardly moves during pressurization and that maximum membrane deformation occurs toward the center of the pars tensa (blue line) and to a lesser degree near the edge of the pars tensa (red line). This observation is confirmed by the 2-D [part (III), (a) to (e)] and 3-D [part (IV), (f) to (j)] visualizations of the full-field membrane height deformation maps, plotted here at discrete intervals [(a) to (e)] during a pressurization cycle. A full video of the eardrum deformation response is included in Video 2. The (semitransparent) colored height deformation map is plotted on top of the input intensity image to illustrate the deformation of the projected line patterns on the moving object surface.

Fig. 7

Video 2 TM height deformation response to dynamic middle ear pressure variation. Height deformation profiles and full-field 2-D and 3-D deformation maps are included in parts (II), (III) and (IV), respectively. The positive $z$-axis is defined toward the observer and the colorbar legend defined in part I applies to parts (III) and (IV), also. (Video 2, 4.74 MB, mp4 [URL: http://dx.doi.org/10.1117/1.JBO.22.1.016008.2]).

Since full-field phase maps are recorded, the user is able to monitor membrane deformation at any desired point on the observable TM surface. This way, the sensitivity of the technique to detect middle ear pressure variation can be maximized by monitoring the response curves of local areas on the TM that deform more than others. As the TM expands up to 1 mm when realistic pressure variations of 0.4 kPa are applied to it, the system’s measurement precision is more than sufficiently high enough to resolve the resulting deformation. Note the asymmetric response curve of the TM topography as a function of induced pressure. The TM displacement induced by middle ear overpressure is larger than for the corresponding underpressure by a factor of almost 2. This effect was previously described by Dirckx and Decraemer³⁰ and can be attributed to the conical shape of the membrane.