A Missile Defense Sensor is a complex optical system, which sits idle for long periods of time, must work with little or no on-board calibration, be used to find and discriminate targets, and guide the kinetic warhead to the target within minutes of launch. A short overview of the Missile Defense problem will be discussed here, as well as, the top-level performance drivers, like Noise Equivalent Irradiance (NEI), Acquisition Range, and Dynamic Range. These top-level parameters influence the choice of optical system, mechanical system, focal plane array (FPA), Read Out Integrated Circuit (ROIC), and cryogenic system. This paper will not only discuss the physics behind the performance of the sensor, but it will also discuss the "art" of optimizing the performance of the sensor given the top level performance parameters. Balancing the sensor sub-systems is key to the sensor’s performance in these highly stressful missions. Top-level performance requirements impact the choice of lower level hardware and requirements. The flow down of requirements to the lower level hardware will be discussed. This flow down directly impacts the FPA, where careful selection of the detector is required. The flow down also influences the ROIC and cooling requirements. The key physics behind the detector and cryogenic system interactions will be discussed, along with the balancing of subsystem performance. Finally, the overall system balance and optimization will be discussed in the context of missile defense sensors and expected performance of the overall kinetic warhead.