We performed numerical simulations and phantom experiments with a conical mirror based fluorescence molecular tomography (FMT) imaging system to optimize its performance. With phantom experiments, we have compared three measurement modes in FMT: the whole surface measurement mode, the transmission mode, and the reflection mode. Our results indicated that the whole surface measurement mode performed the best. Then, we applied two different neutral density (ND) filters to improve the measurement's dynamic range. The benefits from ND filters are not as much as predicted. Finally, with numerical simulations, we have compared two laser excitation patterns: line and point. With the same excitation position number, we found that the line laser excitation had slightly better FMT reconstruction results than the point laser excitation. In the future, we will implement Monte Carlo ray tracing simulations to calculate multiple reflection photons, and create a look-up table accordingly for calibration.