We propose a system that displays a three-dimensional (3D) scene on a 3D display device, consisting of a flat-panel display and a slanted parallax barrier, which can directly manipulate virtual 3D objects in real time. The proposed system consists of a 3D scene display part and a user interaction part. First, we propose a multi-view image interleaving method that considers the distance between a user and a display device for a slanted parallax barrier-based 3D display. Second, we define hand motion parameters for convenient hand motion analysis, and we propose a hand gesture recognition method that uses dynamic time warping (DTW) with specified hand motion parameters for user interaction. Thus, the proposed system makes it possible to see in any position, and viewers can perform event actions without regard to the shape or timing of hand motions.