Position control of multiple objects is one of the most actual problems in various technology areas. For example, in construction area this problem is represented as multi-point deformation control of bearing constructions in order to prevent collapse, in mining – deformation control of lining constructions, in rescue operations – potential victims and sources of ignition location, in transport – traffic control and traffic violations detection, in robotics –traffic control for organized group of robots and many other problems in different areas. Usage of stationary devices for solving these problems is inappropriately due to complex and variable geometry of control areas. In these cases self-organized systems of moving visual sensors is the best solution. This paper presents a concept of scalable visual sensor network with swarm architecture for multiple object pose estimation and real-time tracking. In this article recent developments of distributed measuring systems were reviewed with consequent investigation of advantages and disadvantages of existing systems, whereupon theoretical principles of design of swarming visual sensor network (SVSN) were declared. To measure object coordinates in the world coordinate system using TV-camera intrinsic (focal length, pixel size, principal point position, distortion) and extrinsic (rotation matrix, translation vector) calibration parameters were needed to be determined. Robust camera calibration was a too resource-intensive task for using moving camera. In this situation position of the camera is usually estimated using a visual mark with known parameters. All measurements were performed in markcentered coordinate systems. In this article a general adaptive algorithm of coordinate conversion of devices with various intrinsic parameters was developed. Various network topologies were reviewed. Minimum error in objet tracking was realized by finding the shortest path between object of tracking and bearing sensor, which set global coordinate system. Weight coefficients were determined by experimental researches of system sensors that are represented in this article. Conclusions obtained from this work are the basement for SVSN prototypes production and its future researches.