Energy Recovering Linacs (ERLs) offer an attractive alternative as drivers for light sources as they combine the desirable characteristics of both storage rings (high efficiency) and linear accelerators (superior beam quality). Using superconducting RF technology allows ERLs to operate more efficiently because of the inherent characteristics of SRF linacs, namely that they are high gradient-low impedance structures and their ability to operate in the long pulse or CW regime. We present an overview of the physics challenges encountered in the design and operation of ERL based light sources with particular emphasis on those issues related to SRF technology. These challenges include maximizing a cavity's Qo to increase cryogenic efficiency, maintaining control of the cavity field in the presence of the highest feasible loaded Q and providing adequate damping of the higher-order modes (HOMs). If not sufficiently damped, dipole HOMs can drive the multipass beam breakup (BBU) instability which ERLs are particularly susceptible to. Another challenge involves efficiently extracting the potentially large amounts of HOM power that are generated when a bunch traverses the SRF cavities and which may extend over a high range of frequencies. We present experimental data from the Jefferson Lab FEL Upgrade, a 10 mA ERL light source presently in operation, aimed at addressing some of these issues. We conclude with an outlook towards the future of ERL based light sources.