Most of objects on marine surface are ships and buoys. There are a lot of researches on object recognition and localization for ship, but few studies are focus on marine buoy detection. If marine buoys could be detected in time, most of the damage caused by collision with ships or unmanned vessels will be avoided. We propose a method for automatic detection of marine buoy: by using infrared camera we obtain video stream data in which we select and annotate infrared image samples with buoys, by training those samples we fine-tune a CNN detection model based on MobleNet-SSD which is pre-trained on the Pascal VOC dataset. Any marine buoy could be recognized and localized by our detection method, whose average precision on our test set is up to 90%. We get real-time performance with FPS up to 30 on portable embedded devices like NVIDIA Jetson AGX Xavier and Intel Neural Compute Stick 2.