Colorectal cancer is the second leading cause of cancer deaths in the United States. Identifying and removing premalignant lesions via colonoscopy can significantly reduce colorectal cancer mortality. Unfortunately, the protective value of screening colonoscopy is limited because more than one quarter of clinically-important lesions are missed on average. Most of these lesions are associated with characteristic 3D topographical shapes that appear subtle to a conventional colonoscope. Photometric stereo endoscopy captures this 3D structure but is inherently qualitative due to the unknown working distances from each point of the object to the endoscope. In this work, we use deep learning to estimate the depth from a monocular endoscope camera. Significant amounts of endoscopy data with known depth maps is required for training a convolutional neural network for deep learning. Moreover, this training problem is challenging because the colon texture is patient-specific and cannot be used to efficiently learn depth. To resolve these issues, we developed a photometric stereo endoscopy simulator and generated data with ground truth depths from a virtual, texture-free colon phantom. These data were used to train a deep convolutional neural field network that can estimate the depth for test data with an accuracy of 84%. We use this depth estimate to implement a smart photometric stereo algorithm that reconstructs absolute depth maps. Applying this technique to an in-vivo human colonoscopy video of a single polyp viewed at varying distance, initial results show a reduction in polyp size measurement variation from 15.5% with conventional to 3.4% with smart photometric reconstruction.
Colorectal cancer is the fourth leading cause of cancer deaths worldwide. The detection and removal of premalignant lesions through an endoscopic colonoscopy is the most effective way to reduce colorectal cancer mortality. Unfortunately, conventional colonoscopy has an almost 25% polyp miss rate, in part due to the lack of depth information and contrast of the surface of the colon. Estimating depth using conventional hardware and software methods is challenging in endoscopy due to limited endoscope size and deformable mucosa. In this work, we use a joint deep learning and graphical model-based framework for depth estimation from endoscopy images. Since depth is an inherently continuous property of an object, it can easily be posed as a continuous graphical learning problem. Unlike previous approaches, this method does not require hand-crafted features. Large amounts of augmented data are required to train such a framework. Since there is limited availability of colonoscopy images with ground-truth depth maps and colon texture is highly patient-specific, we generated training images using a synthetic, texture-free colon phantom to train our models. Initial results show that our system can estimate depths for phantom test data with a relative error of 0.164. The resulting depth maps could prove valuable for 3D reconstruction and automated Computer Aided Detection (CAD) to assist in identifying lesions.