A coherent light beam is used to interrogate the focal region within a tissue-mimicking phantom insonified by an ultrasound transducer. The ultrasound-tagged photons exiting from the object carry with them information on local optical path length fluctuations caused by refractive index variations and medium vibration. Through estimation of the force distribution in the focal region of the ultrasound transducer, and solving the forward elastography problem for amplitude of vibration of tissue particles, we observe that the amplitude is directed along the axis of the transducer. It is shown that the focal region interrogated by photons launched along the transducer axis carries phase fluctuations owing to both refractive index variations and particle vibration, whereas the photons launched perpendicular to the transducer axis carry phase fluctuations arising mainly from the refractive index variations, with only smaller contribution from vibration of particles. Monte-Carlo simulations and experiments done on tissue-mimicking phantoms prove that as the storage modulus of the phantom is increased, the detected modulation depth in autocorrelation is reduced, significantly for axial photons and only marginally for the transverse-directed photons. It is observed that the depth of modulation is reduced to a significantly lower and constant value as the storage modulus of the medium is increased. This constant value is found to be the same for both axial and transverse optical interrogation. This proves that the residual modulation depth is owing to refractive index fluctuations alone, which can be subtracted from the overall measured modulation depth, paving the way for a possible quantitative reconstruction of storage modulus. Moreover, since the transverse-directed photons are not significantly affected by storage modulus variations, for a quantitatively accurate read-out of absorption coefficient variation, the interrogating light should be perpendicular to the focusing ultrasound transducer axis.