Rare-earth ions doped in crystals are renowned for their excellent coherence properties and large inhomogeneous broadening, which make them ideal for quantum interfaces with broadband photons. These properties have made them one of the leading technologies in quantum optical memories and a promising candidate for optical-to-microwave conversion. To take advantage of the full bandwidth of the rare-earth ensemble, one must overcome the decoherence of a broadband collective excitation due to inhomogeneous broadening. To this end, techniques based on controllable rephasing, such as atomic frequency comb (AFC) or controlled reversible inhomogeneous broadening (CRIB) memories, have been developed with great success. Recently, an alternative method was proposed to suppress the decoherence of an inhomogeneous ensemble via strong coupling to a cavity, a phenomenon called cavity protection. This technique has been demonstrated in the microwave domain with an NV spin ensemble, but has not been demonstrated in the optical domain. Here, we demonstrate cavity protection in the optical domain at the single photon level using an ensemble of rare earths ions coupled to a nanophotonic resonator. The reduction in decoherence due to the cavity-protection effect enables transfer of ultrafast (~50 GHz) frequency qubits into the collective ion excitation and retrieval with 98.7% fidelity. Building on these results to transfer these excitations to long-lived spin states would enable broadband, on-demand quantum memories. Furthermore, this works compliments the work done coupling rare-earths to superconducting resonators in the microwave regime with potential for applications in optical-to-microwave transducers.