The counting of crop plants is an important basis for the estimation of pitaya yield. Traditional crop monitoring methods are time-consuming and laborious. The pitaya tree, one of the characteristic economic crops in the complex mountain environment, was selected as the research object. Considering the comprehensive factors such as different seasons, cloud shadow shading, crop interplanting, steep terrain, different breeds, and ages of pitaya trees, the quad-rotor unmanned aerial vehicles (UAVs) were used to collect the visible light images of the planting base of pitaya trees in the test area in karst plateau canyon region and the verification area in a complex environment. First, the characteristics of the RGB values of the eight main ground objects in the test areas are analyzed according to the color index, geometric size, and texture characteristics, which show that pitaya plants, weeds, and shrubs have certain interference with each other. Then the excess green index (ExG) of the images is calculated to enhance the vegetation characteristics and separate vegetation and non-vegetation. Gaussian high-pass filter (GHPF) is used to retain the high-frequency information of pitaya plants on the ExG images. After GHPF processing, the DN values of shrub and weeds are reduced, the edges of the area target are enhanced, and the influences of weeds and cloud shadows on plant identification are eliminated. Finally, through field measurement of pitaya plant data and OTSU, grayscale segmentation was performed on the images processed by GHPF and the pitaya plant information was extracted. Combined with the projection area of single pitaya tree, the number of pitaya trees was obtained by area screening method. The target detection percentage of test area A, test area B, and the verification area in complex environment is, respectively, 96.99%, 94.66%, and 94.30%; the quality percentage of them is, respectively, 92.46%, 90.41% and 91.50%; the branching factor of them is, respectively, 0.05, 0.05, and 0.03, proving that the RGB images collected by UAV can be used in the recognition and counting of pitaya trees in complex mountain environments. Our study can provide a reference for the application of low-cost, high-efficiency UAV visible light remote sensing to precision agriculture in complex mountain environment.