Recently, target detection in sea environment such as boat detection has become a popular research topic which is significant for marine vessels monitoring system. Many target detection methods have been widely applied to practical applications such as frame difference, traditional optical flow and background subtraction method. However, the existing target detection methods are not suitable to deal with the complex conditions of sea surface, such as irregular movement of the waves and illumination changes. In this paper, we developed an approach based on vector accumulation of particle motion mainly aiming at eliminating the effects of irregular movement of waves. Our proposed method applies vector accumulation of particle motion to optical flow field to obtain more accurate detection results under complex conditions. Firstly, the traditional optical flow method is used to acquire motion vector of every particle. Furthermore, the vectors of each flow point are abstracted to represent the recording of a fluid element in the flow over a certain period, succeeding is the accumulation of particle vectors. Finally, we calculate the mean of the vector accumulation to eliminate the effects of irregular movement of waves based on the video. Experimental results show the proposed method can gain better performance than traditional optical flow method.