In this paper we introduce the Hankel transform based channel model of Two-Dimensional Optical Storage (TwoDOS) system. Based on this model, the two-dimensional (2D) minimum mean-square error (MMSE) equalizer has been derived and applied to some simple but common cases. The performance of the 2D Viterbi detector (VD) for different partial response (PR) targets in TwoDOS system is investigated. And an easy way of jointly designing the target and equalizer with suitable constraints on the target for 2D data storage systems is proposed by reducing the 2D target into its one-dimensional (1D) form. Finally, systems with different target constraints are compared and the results show that "2D monic constraint" is a reasonable target constraint for TwoDOS system.