Challenges in object tracking such as object deformation, occlusion, and background variations require a robust tracker to ensure accurate object location estimation. To address these issues, we present a Pyramidal Rotation Invariant Features (PRIF) that integrates Gaussian Ringlet Intensity Distribution (GRID) and Fourier Magnitude of Histogram of Oriented Gradients (FMHOG) methods for tracking objects from videos in challenging environments. In this model, we initially partition a reference object region into increasingly ﬁne rectangular grid regions to construct a pyramid. Histograms of local features are then extracted for each level of pyramid. This allows the appearance of a local patch to be captured at multiple levels of detail to make the algorithm insensitive to partial occlusion. Then GRID and magnitude of discrete Fourier transform of the oriented gradient are utilized to achieve a robust rotation invariant feature. The GRID feature creates a weighting scheme to emphasize the object center. In the tracking stage, a Kalman ﬁlter is employed to estimate the center of the object search regions in successive frames. Within the search regions, we use a sliding window technique to extract the PRIF of candidate objects, and then Earth Mover’s Distance (EMD) is used to classify the best matched candidate features with respect to the reference. Our PRIF object tracking algorithm is tested on two challenging Wide Area Motion Imagery (WAMI) datasets, namely Columbus Large Image Format (CLIF) and Large Area Image Recorder (LAIR), to evaluate its robustness. Experimental results show that the proposed PRIF approach yields superior results compared to state-of-the-art feature based object trackers.