A broadband spatial optical filtering system is presented in this paper, which is composed of a Volume Bragg Grating (VBG) and a blazed grating pair. The diffraction efficiency and filtering properties are calculated and simulated by using Fourier diffraction analysis and Coupled Wave Theory. A blazed grating pair and VBG structures are designed and optimized in our simulation. The diffraction efficiency of filtering system shows more than 77.2% during the wavelength period from 953nm to 1153nm, especially 84.1% at the center wavelength. The beam quality is described with near-field modulation (M) and contrast ratio (C). The M of filtering beam are 1.44, 1.49 and 1.55, respectively and the C of filtering beam are 10.1%, 10.2% and 10.5% , respectively and the beam intensity distribution is great improved. The cut-off frequencies of three filtering systems are 1.57mm<sup>-1</sup> , 2.06 mm<sup>-1</sup> and 2.38 mm<sup>-1</sup> , respectively from power spectral density (PSD) curve. It’s clear that the cut-off frequency of filtering system is closely related to the angular selectivity of VBG, and the value of cut-off frequency is decided by VBG’s Half Width at First Zero (HWFZ) and center wavelength.