This paper proposes an incremental shape recovery and registration method based on weak perspective factorization and pairwise point cloud registration. While a camera is moving freely in unknown environments, point features are automatically extracted and tracked. The 3-D position of scene points is then estimated using the tracked features from four consecutive frames and the point set is immediately registered to the global point cloud. The advantage of the proposed incremental method is that the recovery and registration process relies only on past frames for estimating a global shape. Shape recovery and registration are unified into a single framework where all the internal steps are linear and non-iterative. An application can utilize intermediate results that are incrementally computed from past frames and thus the procedure is applicable to on-line interactive vision systems such as augmented reality applications. Experimental results on several real image sequences showing the feasibility of the proposed approach are also provided.