The performance of the classic split-and-merge segmentation algorithm is hampered by its rigid split-and-merge processes, which is insensitive to the image semantics. This paper proposes efficient algorithm and computing structure to optimize the split-and-merge processes by using the optimal dichotomy based on parallel computing. Compared to the common quadtree method, the optimal dichotomy split algorithm is shown to be more adaptive to the image semantics, which means it can avoid excessive split to some degree. We also overcome the problem that the merge iteration process requires too much by diving the image into some fixed width and height sub-images, these sub-images have one pixel wide boarder overlapped to confirm the edge information not lost. Based on the parallel computing model and platform, these sub-images’ edge can be detected within the map procedure rapidly, then we reduce the sub-images’ edges to get the whole final image segment result.