This paper describes a prototype system for identifying and characterizing Multiple Sclerosis (MS) lesions in the brain from magnetic resonance (MR) images. The system is designed to obtain an initial segmentation of each cross-sectional image with low level vision methods, and then derive successive refinements of image subregions through a model-driven approach that correlates relevant information from T1 and T2 images and 3-D information from complementary cross-sections when necessary. The system uses a b-spline surface model of the brain that matches the characteristics of the individual's brain. The normal internal structures of the brain are then scaled proportionately before carrying out the successive refinement operations for the detection of the MS lesions. The low level vision and the solid modeling components of the system have been successfully tested on several hundred images from a number of MR patient studies. The first steps of model fitting have been implemented and show promising results.