To facilitate the clinical diagnosis, surgical planning and after operation follow-up, a computer aided septal defect diagnosis and detection framework is proposed. The framework is consisted of four steps: image registration, flow balance measurement, heart wall tracking and septal defects detection. First, a global smooth constrained localized registration method is employed to register the image; Then, flow balance measurement is employed to determine the unbalance of in and out blood flow, which usually indicate the septal defects in the heart; After that, the wall of the heart is tracked using the same framework used in the registration to improve the efficiency and accuracy; Defects along septal are detected using a Bayesian based information fusion to analyze the profile lines from registered image, difference image and original grey image for the whole sequence (3D+T). The proposed method is tested using gated cardiac MRI, which is a well-established clinical diagnosis method for septal defect detection. Experimental results show that the proposed framework is able to successfully detect the septal defects and provide the visual assistance to the radiologist for further diagnosis. The proposed detection can be widely used in both clinical practice, surgical planning and after operation follow-up. To the best of our knowledge, the work is first such an effort.