Demands for a hydrogen fuel has been increased due to usages as an ecological and alternative energy resource. On the other hand, hydrogen easily causes an explosion above concentrations of 4 % in air, hence hydrogen sensors are need to have rapidity and accuracy for detecting hydrogen. Conventional hydrogen sensors have mainly used palladium (Pd) which is known as a hydrogen detecting material with high sensitivities and selectivity to hydrogen. Generally, Pd absorbs hydrogen in large amounts and forms Pd hydride, moreover, Pd experiences α-β phase transition during volume change of Pd with hydrogen absorption. As a result, the volume change of Pd induces a deterioration which affects time responses and sensitivities of hydrogen sensors. To keep Pd from deteriorating, alloying Pd with metals, such as Au and Ag, has been utilized as preventing Pd from experiencing α-β phase transition. In this paper, we propose a hetero-core optical fiber hydrogen sensor based on surface plasmon resonance (SPR) with multi-layers of Au/Ta2O5/Pd/Au in order to suppress the deterioration of Pd. A few sensors were prepared with the same construction of sensitive film 25-nm Au/ 60-nm Ta2O5/ thicknesses with stacks of annealed 3 double layers of 1.4-nm Pd and 0.6-nm Au or 5-nm pure Pd, and evaluated in terms of the time response and sensitivities. The response times at the 1st and the 15th hydrogen absorption test were experimentally observed to be from 3 s to 6 s for annealed Pd-Au, in contrast, to be from about 16 s to 22 s for pure Pd at 4 % hydrogen concentration, respectively.