Lead-halide perovskites are currently the highest-performing solution-processable semiconductors for solar energy conversion, with record efficiencies rapidly approaching that of the Shockley-Queisser limit for single-junction solar cells. Further progress in the development of lead-halide perovskite solar cells must overcome this limit, which largely stems from the ultrafast relaxation of high-energy hot carriers above the bandedge. In this contribution, we use a highly-specialized pump-push-probe technique to unravel the key parameters which control hot carrier cooling in bulk and nanocrystal (NC) lead bromide perovskites with different material composition, NC diameter and surface treatment. All samples exhibit slower cooling for higher hot carrier densities, which we assign to a phonon bottleneck mechanism. By comparing this density-dependent cooling behavior in the different samples, we find that the weak quantum confinement of electronic states and the surface defects in the NCs play no observable role in the hot carrier relaxation. Meanwhile, in accordance with our previous observations for bulk perovskites, we show that the cation plays a critical role towards carrier cooling in the perovskite NCs, as evidenced by the faster overall cooling in the hybrid FAPbBr3 NCs with respect to the all-inorganic CsPbBr3 NCs. These observations highlight the crucial role of the cations toward the phononic properties of lead-halide perovskites, and further point towards the defect tolerance of these emerging solution-processed semiconductors.