The convolution neural network (CNN) is becoming more and more powerful in many areas such as image classification and speech recognition. Some projects begin to apply it on mobile phones, but often need plenty of time due to the huge amount of computation. This paper uses a deep learning framework named MXNet to realize the forward process on the mobile phone. In order to lower the time it costs, we focus on how to use the other computing device on the chip—the mobile GPU. We choose the OpenCL to offload the most time consuming layer in the CNN—convolution layer to the GPU. Besides that, this paper makes several improvements to achieve better performance and finally the experimental results demonstrate that the forward process only takes half the time in our algorithm. To the best of the authors’ knowledge, this work is the first published implantation of CNN accelerating on mobile GPU.