To assess the health of buildings, maximum inter-story drift angle is recognized as an important indicator. If we have to estimate maximum inter-story drift angle very precisely, we need to install accelerometers on all floors. However, it is not realistic due to the cost. In many methods to estimate the response using small number of accelerometers, the excitation (input) is assumed to be available. However, in some cases, some sensors including the input sensor may not be available. Thus, in this paper, we propose a method for the estimating inter-story drift angle using small number of accelerometers without knowing input information. The proposed method is based on two assumptions. One is that the response is represented by the superposition of the response of only lower modes. The other is that mode vectors and participation factors are available from the structural design model. Based on the assumption, first, we estimate modal frequencies and damping ratios using the subspace method from obtained acceleration data. Second, we decompose observed acceleration data to each mode by solving simultaneous equations using pseudo-inverse matrix. Third, we calculate mode response by focusing on the vibration equation of each mode. It was verified that this method could successfully estimate the modal response as well as the inter-story drift angles.