There is ongoing interest in extending CT from anatomical to functional imaging. Recent successes with dual energy CT, the introduction of energy discriminating x-ray detectors, and novel, target-specific, nanoparticle contrast agents enable functional imaging capabilities via spectral CT. However, many challenges related to radiation dose, photon flux, and sensitivity still must be overcome. Here, we introduce a post-reconstruction algorithm called spectral diffusion that performs a robust material decomposition of spectral CT data in the presence of photon noise to address these challenges. Specifically, we use spectrally joint, piece-wise constant kernel regression and the split Bregman method to iteratively solve for a material decomposition which is gradient sparse, quantitatively accurate, and minimally biased relative to the source data. Spectral diffusion integrates structural information from multiple spectral channels and their corresponding material decompositions within the framework of diffusion-like denoising algorithms. Using a 3D, digital bar phantom and a material sensitivity matrix calibrated for use with a polychromatic x-ray source, we quantify the limits of detectability (CNR = 5) afforded by spectral diffusion in the triple-energy material decomposition of iodine (3.1 mg/mL), gold (0.9 mg/mL), and gadolinium (2.9 mg/mL) concentrations.