Optical methods have been widely utilized in civil engineering deformation/displacement measuring such as robotic total stations (RTS), image/non-contact measurement et al. However, in the practice of measuring in civil engineering, the installation positions of the measuring points and measuring devices were decided usually from human experience. This paper described an optimization method of deciding the installation position in deformation monitoring of multiple points by robotic total stations as an example. The method of resection and polar coordinate was utilized in deformation monitoring, in which two datum points should be installed in immovable location and measuring system could be installed in movable location. The basic errors from optical measuring system are angle errors and distance errors, which combined the systematic measuring errors, which could be reduced by optimizing the installation of the measuring system. This paper presented methods of evaluating the systematic accuracy-functions of the locations of the measuring system and two datum points, the idea of which is from the error analysis in the geodesy field. Then the systematic accuracy could be optimized by optimization methods such as simplex method. The method could be applied in the deformation measurement by RTS or image/non-contact measurement. In most measuring cases by RTS, the problem is a constrained optimization problem with 9 parameters.