Many pedestrian detection research works focused on the improvement of detection performance, without considering the detection speed, making the detection algorithms not applicable for real-world requirement for real-time processing. To explore this problem, we first propose a pre-processing method Hierarchical HOG Matrices to replace the traditional integral histogram of gradients, which stores more data in the pre-processing phase to reduce computation time. A matrix-based detection computation structure is also proposed, which organize the massive data computations in the scanning detection process into matrix operations to optimize the overall speed. We then add multiple instance learning into the fast pedestrian detection algorithm to further enhance its accuracy. Experiments demonstrate that the proposed fast and robust pedestrian detection algorithm based on the multiple instance feature achieves an accuracy comparable to the latest algorithms, with the best speed among the algorithms with an accuracy of the same level.