The histogram of oriented gradients has been proven to be a successful method of object detection, especially for pedestrian detection in images and videos. However, the question of how to make maximal use of color information for gradient calculation has not been thoroughly investigated. We propose a simple yet effective adaption that uses a combination of grayscale-based gradient and color-invariant-based gradients (after Geusebroek et al.) to replace the original gradient definition. Our experiments show that such a combination achieves a 30% reduction in miss rate, using the same experiment setting and the same evaluation criteria as Dalal et al. We have also measured the trade-off between the performance and computational cost by using a more sophisticated quadratic kernel instead of a linear kernel. While it can reduce the miss rate further by 10% to 20%, using a quadratic kernel can take as much as 70 times more running time for the original (Dalal et al. 2006) dataset.