Raman spectroscopy (RS) has demonstrated great potential for in vivo cancer screening; however, the biophysical changes that occur for specific diagnoses remain unclear. We recently developed an inverse biophysical skin cancer model to address this issue. Here, we presented the first demonstration of in vivo melanoma and nonmelanoma skin cancer (NMSC) detection based on this model. We fit the model to our previous clinical dataset and extracted the concentration of eight Raman active components in 100 lesions in 65 patients diagnosed with malignant melanoma (MM), dysplastic nevi (DN), basal cell carcinoma, squamous cell carcinoma, and actinic keratosis. We then used logistic regression and leave-one-lesion-out cross validation to determine the diagnostically relevant model components. Our results showed that the biophysical model captures the diagnostic power of the previously used statistical classification model while also providing the skin’s biophysical composition. In addition, collagen and triolein were the most relevant biomarkers to represent the spectral variances between MM and DN, and between NMSC and normal tissue. Our work demonstrates the ability of RS to reveal the biophysical basis for accurate diagnosis of different skin cancers, which may eventually lead to a reduction in the number of unnecessary excisional skin biopsies performed.