In this paper, we address the problem of decentralized visibility-based target tracking for a team of mobile observers trying to track a team of mobile targets. Based on the results of previous work, we address the problem when the graph that models the communication network within the team of the obsrevers is not complete. We propose a hierarchical approach. At the upper level, each observer is allocated to a target through a local minimum cost matching. At the lower level, each observer computes its navigation strategy based on the results of the single observer-single target problem, thereby, decomposing a large multi-agent problem into several 2- agent problems. Finally, we evaluate the performance of the proposed strategy in simulations and experiments.