The overlapping structures in a chest radiograph can make the detection of pneumothorax difficult. In addition, the visual signs of a pneumothorax, including a fine line at the edge of the lung and a change in texture outside the lung, can be subtle. Some published studies have reported high performances using deep learning for the detection of pneumothorax in chest radiographs using the publicly available ChestX-ray8 dataset. However, at the image input sizes these studies used, 256 x 256 or 224 x 224 pixels, the visual signs of a pneumothorax are typically not visible. In this study, radiographs labeled as pneumothorax in the ChestX-ray8 dataset were interpreted by a radiologist and then confirmed using the radiologist-defined truth from a pneumothorax challenge database. In addition, chest radiographs with and without pneumothorax were obtained from our institution and verified. Therefore, the entire dataset of 5,346 radiographs had truth confirmed by two radiologists. The dataset was used for fine-tuning a VGG19 neural network for the task of detecting pneumothorax in chest radiographs. After fine-tuning was complete, network visualization was performed using Grad-CAM to determine the most influential aspects of the radiograph for the network’s classification. It was found that 67% of Grad-CAM heatmaps for correctly classified pneumothorax cases did not have regions of high influence that overlapped with the actual location of the pneumothorax. Overall, the independent test set yielded an AUC of 0.78 (95% confidence interval: 0.74, 0.82) in the task of distinguishing between radiographs with and without pneumothorax.