Purpose: Surgical training could be improved by automatic detection of workflow steps, and similar applications of image processing. A platform to collect and organize tracking and video data would enable rapid development of image processing solutions for surgical training. The purpose of this research is to demonstrate 3D Slicer / PLUS Toolkit as a platform for automatic labelled data collection and model deployment. Methods: We use PLUS and 3D Slicer to collect a labelled dataset of tools interacting with tissues in simulated hernia repair, comprised of optical tracking data and video data from a camera. To demonstrate the platform, we train a neural network on this data to automatically identify tissues, and the tracking data is used to identify what tool is in use. The solution is deployed with a custom Slicer module. Results: This platform allowed the collection of 128,548 labelled frames, with 98.5% correctly labelled. A CNN was trained on this data and applied to new data with an accuracy of 98%. With minimal code, this model was deployed in 3D Slicer on real-time data at 30fps. Conclusion: We found the 3D Slicer and PLUS Toolkit platform to be a viable platform for collecting labelled training data and deploying a solution that combines automatic video processing and optical tool tracking. We designed an accurate proof-of-concept system to identify tissue-tool interactions with a trained CNN and optical tracking.
PURPOSE: Virtual reality (VR) simulation is an effective training system for medical residents, allowing them to gain and improve upon surgical skills in a realistic environment while also receiving feedback on their performance. Percutaneous nephrolithotomy is the most common surgical treatment for the removal of renal stones. We propose a workflow to generate 3D soft tissue and bone models from computed tomography (CT) images, to be used and validated in a VR nephrolithotomy simulator. METHODS: Venous, delay, non-contrast, and full body CT scans were registered and segmented to generate 3D models of the abdominal organs, skin, and bone. These models were decimated and re-meshed into low-polygon versions while maintaining anatomical accuracy. The models were integrated into a nephrolithotomy simulator with haptic feedback and scoring metrics. Urology surgical experts assessed the simulator and its validity through a questionnaire based on a 5-point Likert scale. RESULTS: The workflow produced soft tissue and bone models from patient CT scans, which were integrated into the simulator. Surgeon responses indicated level 3 and above for face validity and level 4 and above for all other aspects of medical simulation validity: content, construct, and criterion. CONCLUSION: We designed an effective workflow to generate 3D models from CT scans using open source and modelling software. The low resolution of these models allowed integration in a VR simulator for visualization and haptic feedback, while anatomical accuracy was maintained.
PURPOSE: Scoliosis screening is currently only implemented in a few countries due to the lack of a safe and accurate measurement method. Spinal ultrasound is a viable alternative to X-ray, but manual annotation of images is difficult and time consuming. We propose using deep learning through a U-net neural network that takes consecutive images per individual input, as an enhancement over using single images as input for the neural network.
METHODS: Ultrasound data was collected from nine healthy volunteers. Images were manually segmented. To accommodate for consecutive input images, the ultrasound images were exported along with previous images stacked to serve as input for a modified U-net. Resulting output segmentations were evaluated based on the percentage of true negative and true positive pixel predictions.
RESULTS: After comparing the single to five-image input arrays, the three-image input had the best performance in terms of true positive value. The single input and three-input images were then further tested. The single image input neural network had a true negative rate of 99.79%, and a true positive rate of 63.56%. The three-image input neural network had a true negative rate of 99.75%, and a true positive rate of 66.64%.
CONCLUSION: The three-image input network outperformed the single input network in terms of the true positive rate by 3.08%. These findings suggest that using two additional input images consecutively preceding the original image assist the neural network in making more accurate predictions.
PURPOSE: In radiation therapy treatment planning systems, structures are represented as parallel 2D contours. For treatment planning algorithms, structures must be converted into labelmap (i.e. 3D image denoting structure inside/outside) representations. This is often done by triangulated a surface from contours, which is converted into a binary labelmap. This surface to binary labelmap conversion can cause large errors in small structures. Binary labelmaps are often represented using one byte per voxel, meaning a large amount of memory is unused. Our goal is to develop a fractional labelmap representation containing non-binary values, allowing more information to be stored in the same amount of memory. METHODS: We implemented an algorithm in 3D Slicer, which converts surfaces to fractional labelmaps by creating 216 binary labelmaps, changing the labelmap origin on each iteration. The binary labelmap values are summed to create the fractional labelmap. In addition, an algorithm is implemented in the SlicerRT toolkit that calculates dose volume histograms (DVH) using fractional labelmaps. RESULTS: We found that with manually segmented RANDO head and neck structures, fractional labelmaps represented structure volume up to 19.07% (average 6.81%) more accurately than binary labelmaps, while occupying the same amount of memory. When compared to baseline DVH from treatment planning software, DVH from fractional labelmaps had agreement acceptance percent (1% ΔD, 1% ΔV) up to 57.46% higher (average 4.33%) than DVH from binary labelmaps. CONCLUSION: Fractional labelmaps promise to be an effective method for structure representation, allowing considerably more information to be stored in the same amount of memory.
PURPOSE: In radiotherapy treatment planning systems, structures of interest such as targets and organs at risk are stored as 2D contours on evenly spaced planes. In order to be used in various algorithms, contours must be converted into binary labelmap volumes using voxelization. The voxelization process results in lost information, which has little effect on the volume of large structures, but has significant impact on small structures, which contain few voxels. Volume differences for segmented structures affects metrics such as dose volume histograms (DVH), which are used for treatment planning. Our goal is to evaluate the impact of voxelization on segmented structures, as well as how factors like voxel size affects metrics, such as DVH.
METHODS: We create a series of implicit functions, which represent simulated structures. These structures are sampled at varying resolutions, and compared to labelmaps with high sub-millimeter resolutions. We generate DVH and evaluate voxelization error for the same structures at different resolutions by calculating the agreement acceptance percentage between the DVH.
RESULTS: We implemented tools for analysis as modules in the SlicerRT toolkit based on the 3D Slicer platform. We found that there were large DVH variation from the baseline for small structures or for structures located in regions with a high dose gradient, potentially leading to the creation of suboptimal treatment plans.
CONCLUSION: This work demonstrates that labelmap and dose volume voxel size is an important factor in DVH accuracy, which must be accounted for in order to ensure the development of accurate treatment plans.
Segmented structures such as targets or organs at risk are typically stored as 2D contours contained on evenly spaced cross sectional images (slices). Contour interpolation algorithms are implemented in radiation oncology treatment planning software to turn 2D contours into a 3D surface, however the results differ between algorithms, causing discrepancies in analysis. Our goal was to create an accurate and consistent contour interpolation algorithm that can handle issues such as keyhole contours, rapid changes, and branching. This was primarily motivated by radiation therapy research using the open source SlicerRT extension for the 3D Slicer platform. The implemented algorithm triangulates the mesh by minimizing the length of edges spanning the contours with dynamic programming. The first step in the algorithm is removing keyholes from contours. Correspondence is then found between contour layers and branching patterns are determined. The final step is triangulating the contours and sealing the external contours. The algorithm was tested on contours segmented on computed tomography (CT) images. Some cases such as inner contours, rapid changes in contour size, and branching were handled well by the algorithm when encountered individually. There were some special cases in which the simultaneous occurrence of several of these problems in the same location could cause the algorithm to produce suboptimal mesh. An open source contour interpolation algorithm was implemented in SlicerRT for reconstructing surfaces from planar contours. The implemented algorithm was able to generate qualitatively good 3D mesh from the set of 2D contours for most tested structures.