The intrinsic broadband and ultrafast photoresponse of graphene has been extensively studied in recent years, promising the new generation of photodetectors covering the unprecedentedly broad spectrum from THz to near-infrared. It has been demonstrated that the broadband and ultrafast photocurrent generation takes place at the graphene-metal interface with contribution from both photo-thermoelectric and photovoltaic effects, stemming from the efficient generation of hot carriers. Although the hot carrier lifetime is of key importance for their efficient extraction, the dynamics of carrier cooling is still far from being completely understood. So far, two fundamentally different scattering mechanisms have been suggested to dominate in graphene: the momentum-conserved collisions with the high-energy optical phonons, and the disorder-driven supercollisions with the acoustic phonons. However, the co-existing relaxation via both optical and acoustic phonons has not been considered, hindering the interpretation of different experiments within a single physical model. In our work, we discuss the non-uniform graphene properties in the graphene-metal photodetectors, and demonstrate that different cooling mechanisms equally contribute to the process due to the presence of the photocurrentgenerating interface defect. Noting the overlooked role of the metal contact in cooling dynamics, we show that the purity of graphene employed for photodetection is of less importance for the relaxation dynamics compared to the contact area in terms of introduced system disorder. Further, we show that the transient photo-thermoelectric response, so far attributed exclusively to supercollisions, can be predicted by considering the contribution from both relaxation pathways: normal and supercollision scattering of hot carriers.