High peak power picosecond laser ablation of silicon draws great attention in solar cell manufacture,laser optoelectric countermeasure applications, eta. This paper reports the damage process of ultrafast lasers interaction with silicon,which is based on Two-Temperature Model(TTM) and 1-on-1 damage threshold test method. Pulsed laser caused damage manifests in several ways, such as heat damage, mechanical effect and even eletrical effect. In this paper, a modified Two Temperature Model is applied in ultrashort laser interaction with silicon.The traditional Two-Temperature Model methods is proposed by Anismov in 1970s to calculate the interaction between ultrafast laser with metals, which is composed of free electrons and lattice. Beyond the carrier and lattice temperture model, an additional excited term and Auger recombination term of carriers is taken into account in this modified Two-Temperature Model model to reflect the characteristics in semicondutors. Under the same pulse-duration condition, the damage threshold is found to be 161 mJ/cm2 and a characteritic double-peak shape shows up. As the pulse energy density rises from 50mJ/cm2 to 161 mJ/cm2, the difference between carrier and lattice temperature steps down proportionally.Also,a detailed interaction process between photon-electron and electron-phonon is discussed. Electron and lattice temperature evolutes distinctly different, while the former is much higher than the latter until heat tranfer finished at 200 picoseconds. Two-peak feature of electron temperature is also identified. As the pulse duration increases from 20 picosecond to 60 picosecond, the he difference between carrier and lattice temperature steps down significantly. The calculated damage threshold does not change fundamentally, remaining approximately 0.16J/cm2. Also, the damage mechanism is found to be thermal heating with the pulse width between 20 and 60 picoseconds at threshold fluences which is identical to experiment test result. This research is valuable to laser applications and/or laser shielding applications.