Motion estimation is one of the fundamental problems in digital video processing. One of the most notable approaches of motion estimation is based on the estimation of a measure of the change of image brightness in the frame sequence commonly referred to as optical flow. The classical approaches for finding optical flow have many drawbacks. The numerical methods or least square methods for solving optical flow constrains are susceptible to errors in the cases of occlusion and of noise. Two moving objects having common border causes confliction in the velocities, and taking their averages yields a less satisfactory optical flow estimation. The wrong detection of moving boundary, as motion is usually not homogeneous and the inexact contour measurements of moving objects are the other problems of optical flow methods. Therefore, information such as color and edges along with optical flow has been used in the literature. Further, the classical methods need lot of calculations and computations for optical flow measurements. In this paper, we proposed a method, which is very fast and gives better moving information of the objects in the image sequences. The possible locations of moving objects are found first, and then we apply the Hough Transform only on the detected moving regions to find the optical flow vectors for those regions only. So we save lot of time for not finding optical flow for the still or background parts in the image sequences. The new Boolean based edge detection is applied on the two consecutive input images, and then the differential edge image of the resulting two edge maps is found. A mask for detecting the moving regions is made by dilating the differential edge image. After getting the moving regions in the image sequence with the help of the mask obtained already, we use the Hough Transform and voting accumulation methods for solving optical flow constraint equations. The voting based Hough transform avoids the errors associated with least squares techniques. Calculation of a large number of points along the constraint line is also avoided by using the transformed slope-intercept parameter domain. The simulation results show that the proposed method is very effective for extracting optical flow vectors and hence tracking moving objects in the images.