We have designed and implemented a computer-based system that permits rapid acquisition of digital medical images, multi- modality registration and segmentation, and 3D planning of minimally invasive neurosurgical procedures. The system, known as Netra, is optimized for real-time planning: imaging, pre- processing and planning are performed on the morning of surgery in clinically useful times. We have tested the system on procedures such as needle biopsies, depth electrode placements and craniectomies for arteriovenous malformations, aneurysms and tumors. We describe in this paper the core algorithms of our system, and discuss issues related to implementation, validation and user acceptance. We focus on techniques for physician interaction that encourage active participation by the surgeon as principal operator of the visualization and planning system.