In the geospatial data processing, a large number of mathematical models are nonlinear models. The observation equations are of strong nonlinearity and sensitivity to the initial value point of the series expansion. This paper proposes a regularization homotopy improved algorithm which is base on regularization method and homotopy continuation idea. This algorithm constructs regularization homotopy function by adding a stable functional to make nonlinear least square ill-posed problem into optimization problem. The iterative formula is derived by adopting the strategy of<i> f (x)</i> linearization, linking least square principle and introducing step size factor λ in the paper. Finally the calculation results of classical nonlinear least square problem show that regularization homotopy improved algorithm not only low dependence on initial value, but also make small fluctuation in the iterative process, and the solution is stable relatively. The method is correctly and applicable.