In the traditional 3D measurement systems based on phase shifted structured light, the structured light is usually generated by raster projection. But the optical system by using this method is a little big in size for the endoscope. This paper proposes a method for generating structured light based on bundled fibers, which can solve this problem. However, it was found in experiments that the structured light projected by this method is deviated greatly from the standard sinusoidal fringe. In order to compensate for the error caused by the unsatisfactory structural light, we propose a calibration algorithm. To verify the phase compensation algorithm, we had built an endoscope system using the arranged fiber bundles. On the wedge objects with different tilt angles, the measurement error of the phase is improved more than 30% after using phase compensation algorithm. Then, the plaster model with complex surface was used in the experiment. The result indicated that the measured shape of the model is more precise after phase compensation. It is shown that the proposed phase compensation algorithm can effectively correct the error of phase measurement caused by non-sinusoidal structured light in the endoscope system.