Modern medical image techniques, such as magnetic resonance image (MRI) or x-ray computed tomography provide three dimensional images of internal structures of the body, usually by means of a stack of tomographic images. The first stage in the automatic analysis of such data is 3-D edge detection1,2 which provides points corresponding to the boundaries of the surfaces forming the 3-D structure. The next stage is to characterize the local geometry of these surfaces in order to extract points or lines on which registration and/or tracking procedures can rely.3,4,5,6 This paper presents a pipeline of processes which define a hierarchical description of the second order differential characteristics of the surfaces. The focus is on the theoretical coherence of these levels of representation. Using uncertainty, a link is established between the edge detection and the local surface approximation by addressing the uncertainties inherent to edge detection in 2-D or 3-D images; and how to incorporate these uncertainties into the computation of local geometric models. In particular, calculate the uncertainty of edge location, direction, and magnitude for the 3-D Deriche operator is calculated.1,2 Statistical results are then used as a solid theoretical foundation on which to base subsequent computations, such as the determination of local surface curvature using local geometric models for surface segmentation. From the local fitting, for each edge point the mean and Gaussian curvature, principal curvatures and directions, curvature singularities, lines of curvature singularities, and covariance matrices defining the uncertainties are calculated. Experimental results for real data using two 3-D scanner images of the same organ taken at different positions demonstrate the stability of the mean and Gaussian curvatures. Experimental results for real data showing the determination of local curvature extremes of surfaces extracted from MR images are presented.