For visual tracking with UAV, the non-rigid body change of target usually results in the accumulation of errors and decline of tracking precision. In view of this problem, a target regression tracking algorithm based on convolutional neural network is proposed. Firstly, we use the Siamese convolutional neural network to extract features which used as the input of tracker based on self-adapted scale kernel correlation filters. Then, in order to cope with the cumulative errors caused by the change of target form, a target regression network is designed to refine the location. Using the refined location to extract sample and update the filter parameters of tracker can prevent tracker from being polluted. The experimental results show that the algorithm has high tracking precision as well as fast speed compared to the state-of-the-art tracking algorithms, especially with the ability to deal with the non-rigid body change of target.