We propose a method to classify multiple similar actions which are contained in human behaviors by considering a weak-constrained order of “actions”. The proposed method regards the human behavior as a combination of “action” patterns which have order constrained weakly. In this method, actions are classified by using not only image features but also consistency of transitions between an action and next action. By considering such an action transition, our method can recognize human behavior even if image features of different action are similar to each other. Based on this idea, we have improved the previous HMM-based algorithm effectively. Through some experiments using test image sequences of human behavior appeared in a bathroom, we have confirmed that the average classification success rate is 97 %, which is about 53 % higher than the previous method.