Positron emission tomography (PET) is a widely used molecular imaging modality for various clinical applications. With Magnetic Resonance Imaging (MRI) providing anatomical information, simultaneous PET/MR reduces the radiation risk. Both improved hardware and algorithms have been developed to further reduce the amount of radiotracer dosage, but these methods are not yet applied to very low dose. Here, we propose a Deep Learning based method to enable ultra-low-dose PET denoising with multi-contrast information from simultaneous MRI. Methods:The method is implemented to denoise 18F-fluorodeoxyglucose (FDG) brain PET images from low-dose images with 200-fold dose reduction through undersampling, and evaluated for glioblastoma (GBM) patients. Comprehensive quantitative and qualitative evaluations were conducted to verify the performance and clinical applicability of the proposed method, including quantitative accuracy evaluation, visual quality evaluation, reader study with manual tumor segmentation to evaluate the diagnostic quality. Results:The results demonstrate that the proposed method achieves superior results in performance and efficiency comparing with the state-of-art denoising methods. Conclusion:Though reconstructed from scans with only 0.5% of the standard dose, the denoised ultra-low-dose PET images deliver similar visual quality and diagnostic information as the standard-dose PET images. By combining PET and MR information, the proposed Deep Learning based method improves image quality of ultra-low-dose PET, preserves diagnostic quality, and potentially enables much safer, faster, and more cost-effective PET/MR studies.