Machine vision measuring methods are increasingly used in gear measurement. Due to the small size of the fine-pitch gear, the contact measurement method has problems such as difficulty in alignment and low measurement efficiency. It is a better technical solution to measuring fine-pitch gears by machine vision measurement. Compared with the gear measurement center, the machine vision measuring method has the advantages of high efficiency and small measuring force. Gear alignment is relatively simple with the machine vision measuring method. Furthermore, multiple fine-pitch gears can be measured simultaneously, improving the measurement efficiency of fine-pitch gears. Therefore, the machine vision measuring method is particularly suitable for online measurement. However, there is no widely accepted method for the determination of the center position of the gear and the evaluation of pitch deviation in the fine-pitch gear measurement based on machine vision. In this paper, The global least squares fitting algorithm for obtaining centers of each fine-pitch gear is proposed. The experimental results show that the obtained gear center coordinates are more accurate than the gear center coordinates obtained by conventional edge detection algorithm and the centroid method. Firstly, the tip circle, root circle and gear center of the acquired gear image are calculated. Secondly, the Canny algorithm is used to obtain the gear profile. Then, data points on each tooth profile surface are obtained by classifying the gear profile. The least squares arc fitting is performed by using the data points near the reference circle on the tooth profile to obtain the intersection of the fitted arc and the reference circle. Finally, by comparing the measured value with the standard value, a single pitch deviation of the gear can be obtained. The experimental results show that the gear pitch deviation calculated by the tooth profile fitting circle is accurate and the measurement speed is fast. The measuring time of a typical gear is about 15s.