The development of energy-resolving photon-counting detectors for medical x-ray imaging is attracting considerable attention. Since the image quality can be degraded by different nonidealities such as charge sharing, Compton scatter and fluorescence, there is a need for developing performance metrics in order to compare and optimize detector designs. For conventional, non-energy-resolving detectors, this is commonly done using the linear-systems-theory framework, in which the detector performance is described by noise-equivalent quanta (NEQ) and detective quantum efficiency (DQE) as functions of spatial frequency. However, these metrics do not take the energy-resolving capabilities of multibin photon-counting detectors into account. In this work, we present a unified mathematical framework for quantifying the performance of energy-resolving detectors. We show that the NEQ and DQE can be generalized into matrix-valued quantities, which describe the detector performance for detection tasks with both spatial and energy dependence. With this framework, a small number of simple measurements or simulations are sufficient to compute the dose efficiency of a detector design for any imaging task, taking the effects of detector nonidealities on spatial and energy resolution into account. We further demonstrate that the same framework also can be used for assessing material quantification performance, thereby extending the commonly used performance metrics based on the Cramér-Rao lower bound to spatial-frequency-dependent tasks. The usefulness of the proposed framework is demonstrated using simulations of charge sharing and fluorescence in a CdTe detector.