For patients with early-stage breast cancer, the axillary lymph node (ALN) metastasis status is one of the important indicators in breast cancer staging and prognosis. In this study, a computer-aided prediction (CAP) system based on the ultrasound image using the deep learning method to determine the ALN status in breast cancer. In this study, the US imaging database contained 153 malignant tumor images which confirmed by histologically examine, and either SNB or ALND confirmed the axillary metastasis status. The Mask R-CNN method is used to indicate the tumor location and extract the tumor region. After the tumor region segmentation process, we obtained the surrounding tissue region (1 mm and 2 mm), which might include implicit information of the tumor metastasis mechanism. Finally, the convolution neural network (CNN)-based classifier is used to predict the ALN metastasis status using segmented images. In the experiments, the results show that the combined region (tumor with 2 mm surrounding tissue) image has the highest predict performance. The accuracy, sensitivity, specificity, and the area index (Az value) under the receiver operating characteristic (ROC) curve for the CAP system were 77.12%, 66.10%, 84.04%, and 0.7592 for using combined region (tumor with 2 mm surrounding tissue) images. These results indicated that the proposed CAP system can be helpful to determine the ALN status in patients with early-stage breast cancer.