In optical integrated circuits (OICs), inverted tapered spot-size converters (SSC) provide a high coupling efficiency between a silicon nanowire and an optical fiber. However, to reduce the packaging costs of these OICs, it is beneficial to use a SSC with a mode field diameter that matches to that of a conventional single-mode fiber (SMF). Such a SSC offers high alignment tolerance for butt-coupling with a SMF, without the need for a specialized lensed or tapered fiber. In this work, we propose a novel SSC which is not only broadband but also polarization insensitive. The proposed SSC is composed of a stack of Si3N4/SiO2 layers deposited on top of a silicon nanowire. The most optimal modal overlap of our SSC with a conventional SMF of radius 4.2 μm showed 94% and 99%, for TE and TM polarization, respectively. This multilayer stack is tapered along the propagation direction to transfer power to a tapered silicon nanowire. We have studied the adiabatic transfer of power by optimizing the taper lengths such that the minimum loss can be achieved for both polarizations. Our design demonstrates a record low overall coupling loss (including losses due to mode overlap, tapered design, and reflection) of 0.71 dB (for TE) and 0.48 dB (for TM) between a conventional SMF and the silicon nanowire at the wavelength of 1.55 μm. The variation in coupling loss due to the tapered design is less than 0.6 dB over the wavelength range from 1.5 μm to 1.6 μm.