When fringe projection profilometry is used for measuring rough/textured surfaces, the fidelity of the measurement is subject to the spatial frequency response. The instrument transfer function (ITF) is one appealing approach to characterize this property. The foundation of ITF analysis is based on the linear theory; only linear systems are appropriate for ITF analysis. A fringe projection system is intrinsically nonlinear, but it can be approximated as a linear system when certain conditions are met. Here we investigate the linear conditions of a custom fringe projection system designed for an additive manufacturing application. The applicability of ITF is discussed through mathematical analysis and simulations.