Integration of electro-optical and radar generated tracks is critical for identifying accurate time and space position information in target tracking and providing a single integrated picture (SIP) of the dynamic situation. This paper proposes a new, robust, real-time algorithm to (i) correctly correlate data from several sensors and the existing system track, (ii) improve target tracking accuracy and (iii) identify when the data represent new tracks. The proposed algorithm uses metric data, linear, and area features extracted from optical and radar images. The major novelty of the algorithm is in use of robust and affine invariant structural relations built on the features for accurate correlation. These features are combined with intelligent adaptation of Kalman filter using Neural Networks. A proposed measure of confidence with the correlation decision is based on both structural and metric similarities of tracks to estimate both bias and random errors. The similarities are based on concepts from the abstract algebraic systems, generalized Gauss-Markov stochastic processes, and Kalman filters for n-dimensional time series that explicitly model measurement dependence on k previous measurements, M(t/t-1,t-2,...,t-k). These techniques are naturally combined with the hierarchical matching approach to increase the overall track accuracy. The proposed approach and algorithm for track correlation/matching is suitable for both centralized and distributed computing architecture.