Interfaces with subtle difference in atomic and electronic structures in perovskite ABO3 heterostructures often yield intriguingly different properties, yet their exact roles remain elusive. In this article, we report an integrated study of unusual transport, magnetic, and structural properties of Pr0.67Sr0.33MnO3 (PSMO) films and La0.67Sr0.33MnO3 (LSMO) films of various thicknesses on SrTiO3 (STO) substrate. In particular, using atomically resolved imaging and electron energy-loss spectroscopy (EELS), we measured interface related local lattice distortion, BO6 octahedral rotation and cation-anion displacement induced polarization. In the very thin PSMO film, an unexpected interface-induced ferromagnetic polaronic insulator phase was observed during the cubic-to-tetragonal phase transition of the substrate STO, due to the enhanced electron-phonon interaction and atomic disorder in the film. On the other hand, for the very thin LSMO films we observed a remarkably deep polarization in non-ferroelectric STO substrate near the interface. Combining the experimental results with first principles calculations, we propose that the observed deep polarization is induced by an electric field originating from oxygen vacancies that extend beyond a dozen unit-cells from the interface, thus providing important evidence of the role of defects in the emergent interface properties of transition metal oxides.