The chest wall underneath the breast tissue affects near-infrared (NIR) diffusive waves measured with reflection geometry. With the assistance of a co-registered ultrasound, the depth and the tilting angle of the chest wall can be determined and are used to model the breast as a two-layer medium. Finite element method (FEM) is suitable for modeling complex boundary conditions and is adapted to model the breast tissue and chest wall. Four parameters of bulk absorption and reduced scattering coefficients of these two layers are estimated and used for imaging reconstruction. Using a two-layer model, we have systematically investigated the effect of the chest wall on breast lesion reconstruction. Results have shown that chest-wall depth, titling angle, and difference between optical properties of two layers of lesion and reference sites affect the lesion reconstruction differently. Our analysis will be valuable and informative to researchers who are using reflectance geometry for breast imaging. The analysis can also provide guidelines for imaging operators to minimize image artifacts and to produce the best reconstruction results.