Image segmentation is an important application in computer vision. Nowadays, image segmentation of infrared image has not gain as much attention as image segmentation of visible light image. But this application is very useful. For example, searching and tracking targets with infrared search and track system (IRST) has been widely used these days due to its special passive mode. So it can be used as a kind of supplementary equipment for radar. Infrared image segmentation can help computers identify backgrounds of the image, and help it automatically adjust the related parameters for the next work, such as targets recognition or targets detection.<p> </p> Our work proposed a new image segmentation method for infrared image using histogram of oriented gradients (HOG) feature and kernel extreme learning machine (kernel ELM). HOG are feature descriptors which can be used in computer vision and image processing for the purpose of object detection. In this paper, we extract HOG feature of infrared image, and use this feature as the basis for classification. After having feature, we use kernel extreme learning machine to do the segmentation. Kernel extreme learning machine has shown many excellent characteristics in classification. By testing our algorithm proposed in our paper, we demonstrated that our algorithm is effective and feasible.