In this work, plastic wires buried in dry, damp and wet soils are being detected from ground penetrating radar (GPR) images. Such detection is hard, mainly due to three facts: (1) detection of buried targets made of different materials but of the same shape is difficult from GPR images as their signatures look very similar; (2) the same object buried in different soils shows different signatures in a GPR image; and (3) obtaining GPR data in the millions range is not a viable option because of the difficulties in data collection. Therefore in this work, first, domain adaptation (DA) is used to bring the information from previously trained deep learning models on standard image processing tasks into the GPR domain. It is shown that with DA, high classification rates can be achieved even with small GPR datasets, and that these rates surpass the classification rates achieved by convolutional neural networks (CNNs). However, detecting the targets in different soils still remains a problem. Therefore, secondly, a multi-task CNN is proposed, in which, soil and target classification are stitched together. In doing so, our customized classifier detects targets according to soil type, and results in superior classification rates. To the best of our knowledge, we are the first group to use multi-task learning for buried target detection with GPR.