Correlation filter (CF) has been widely used in visual tracking tasks due to its simplicity and high efficiency. However, the conventional CF-based trackers suffer from boundary effects and object scale changes. To address these problems, we propose an adaptive spatially regularized CF tracking method via level set segmentation (LSS). First, a fast LSS method based on region and boundary information is proposed to accurately estimate the object region. Then an adaptive spatial reliability map is constructed using the estimated object region to regularize the CF, which can significantly mitigate the boundary effects. Meanwhile, the estimated object region is also utilized to construct a series of candidate scales whose aspect ratio is variable, and a novel scale variation (SV) constraint term is proposed to restrain the abnormal scale change. Finally, the position and scale of the object is estimated by maximizing the final response, which combines the CF response, and the histogram response is calculated quickly in frequency domain with the SV constraint term. Experimental results on recent visual tracking benchmark datasets illustrate that the tracking capability of the proposed method is competitive against several state-of-the-art trackers.