Cancer recognition is the prerequisite to determine appropriate treatment. This paper focuses on the semantic segmentation task of microvascular morphological types on narrowband images to aid clinical examination of esophageal cancer. The most challenge for semantic segmentation is incomplete-labeling. Our key insight is to build fully convolutional networks (FCNs) with double-label to make pixel-wise predictions. The roi-label indicating ROIs (region of interest) is introduced as extra constraint to guild feature learning. Trained end-to-end, the FCN model with two target jointly optimizes both segmentation of sem-label (semantic label) and segmentation of roi-label within the framework of self-transfer learning based on multi-task learning theory. The learning representation ability of shared convolutional networks for sem-label is improved with support of roi-label via achieving a better understanding of information outside the ROIs. Our best FCN model gives satisfactory segmentation result with mean IU up to 77.8% (pixel accuracy > 90%). The results show that the proposed approach is able to assist clinical diagnosis to a certain extent.