There are directional textures in the residual blocks in H.264. There exists such a phenomenon in which the energy assembles to the upper side of the block after horizontal transformation when the horizontal texture is strong and vertical texture is smooth, and assembles to the left side when the contrary is the case. Based on this phenomenon, an adaptive horizontal and vertical transform skip scheme is proposed in this paper. The thresholds for horizontal and vertical transforms skip mode are presented based on the gradients of the residual block. A modified quantization method is deduced according to the horizontal and vertical transforms skip modes. In addition, three scan modes are adopted based on the energy concentration directions. The simulation results show that the proposed scheme achieves 4.51% to 7.70% total bits saving, and 0.01 to 0.12 dB improvement in peak signal noise ratio without obvious simulation time increment, compared with JM18.1.