Bi-parametric MRI (bpMRI: T2W MRI and Apparent Diffusion Coefficient maps (ADC) derived from diffusion weighted imaging) is increasingly being used to characterize prostate cancer (PCa). However, inter- and intrareader variability hinders interpretation of MRI. Deep learning networks may aid in PCa characterization and may allow for non-invasively distinguishing clinically significant (csPCa: GGG<1) and insignificant (ciPCa: GGG=1) PCa. Recent studies have shown that signatures from peri-tumoral (PT) region on imaging add significant value to those from intra-tumoral (IT) region for disease detection and characterization. In this work, we present a multi-sequence multi-instance learning convolutional neural network trained using 2D patches extracted from PCa regions of interest (ROIs) on prostate bpMRI to distinguish csPCa and ciPCa. The trained classifier is used to extract pooled features from both the IT and PT ROIs, which are then used to train a random forest classifier to distinguish csPCa and ciPCa. We train and test our models using patient studies from two different institutions (n=298) with GGG obtained either from post-surgical specimens or biopsies. Model built using IT (<i>D<sub>IT</sub></i>) and PT (<i>D<sub>PT</sub></i>) deep features alone resulted in an area under the curve (AUC) of 0.83 and 0.73 respectively, while models computed from IT (<i>R<sub>IT</sub></i>) and PT (<i>R<sub>PT</sub></i>) radiomic features resulted in an AUC of 0.77 and 0.75 respectively. The models <i>D<sub>IP</sub></i> and <i>R<sub>IP</sub></i> trained on combination of IT and PT deep features and radiomic features resulted in an AUC of 0.86 and 0.80 respectively. In both cases, we observe that combining IT and PT features helps in improving the overall classifier performance in distinguishing csPCa and ciPCa.