Significance: The diagnosis of prostate cancer (PCa) and focal treatment by brachytherapy are limited by the lack of precise intraoperative information to target tumors during biopsy collection and radiation seed placement. Image-guidance techniques could improve the safety and diagnostic yield of biopsy collection as well as increase the efficacy of radiotherapy.

Aim: To estimate the accuracy of PCa detection using in situ Raman spectroscopy (RS) in a pilot in-human clinical study and assess biochemical differences between in vivo and ex vivo measurements.

Approach: A new miniature RS fiber-optics system equipped with an electromagnetic (EM) tracker was guided by trans-rectal ultrasound-guided imaging, fused with preoperative magnetic resonance imaging to acquire 49 spectra in situ (in vivo) from 18 PCa patients. In addition, 179 spectra were acquired ex vivo in fresh prostate samples from 14 patients who underwent radical prostatectomy. Two machine-learning models were trained to discriminate cancer from normal prostate tissue from both in situ and ex vivo datasets.

Results: A support vector machine (SVM) model was trained on the in situ dataset and its performance was evaluated using leave-one-patient-out cross validation from 28 normal prostate measurements and 21 in-tumor measurements. The model performed at 86% sensitivity and 72% specificity. Similarly, an SVM model was trained with the ex vivo dataset from 152 normal prostate measurements and 27 tumor measurements showing reduced cancer detection performance mostly attributable to spatial registration inaccuracies between probe measurements and histology assessment. A qualitative comparison between in situ and ex vivo measurements demonstrated a one-to-one correspondence and similar ratios between the main Raman bands (e.g., amide I-II bands, phenylalanine).

Conclusions: PCa detection can be achieved using RS and machine learning models for image-guidance applications using in situ measurements during prostate biopsy procedures.