A large-scale optical matrix switch is generally composed of a permutation of switching units connected by a network of optical signal paths. In this work, a configuration of optical matrix switches with flexible switching units and Banyan networks is proposed. With Banyan networks, silica-based waveguides, and two types of switching units, the insertion loss of the optical matrix switch configuration is extensively studied for serial scales from 2×2 to 64×64. Typically for an 8×8 optical matrix switch, the insertion losses are achieved 3.0 dB for a rearrangeably nonblocking configuration with two-state 2×2 switching units, and 4.7 dB for a strictly nonblocking configuration with three-state 2×2 switching units when the intersection angle is designed as 30 deg. Even at the released intersection angle of 20 deg, the insertion losses of 4.0 and 5.8 dB can be achieved for the rearrangeably nonblocking and strictly nonblocking configurations, respectively. In addition, the blocking property and the scalability of the large-scale matrix optical switches with this regime are analyzed.