This paper proposes a novel human action recognition framework named Hidden Markov Model (HMM) based Hybrid Event Probability Sequence (HEPS), which can recognize unlabeled actions from videos. First, motion trajectories are effectively extracted using the centers of moving objects. Secondly, the HEPS is constructed using the trajectories and represents different human actions. Finally, the improved Particle Swarm Optimization (PSO) with inertia weight is introduced to recognize human actions using HMM. The proposed methods are evaluated on UCF Human Action Dataset and achieve 76.67% accurate rate. The comparative experiments results demonstrate that the HMM got superior results with HEPS and PSO.