The proposed MOSAIC first-generation instrument for the ELT is a multi-object spectrograph utilising a combined MOAO and GLAO system. With 8 separate wavefront sensors (4 LGS and 4 NGS), and 10 separate deformable mirrors, in addition to the ELT M4 mirror, MOSAIC represents one of the most challenging ELT instruments for real-time control, using a total of approximately 65,000 slope measurements to control approximately 26,000 actuators with a 250 Hz LGS frame rate. The proposed modular design of real-time control system to be used with MOSAIC is presented. This is based on the Durham AO Real-time Controller (DARC), and uses 12x Intel Xeon Phi nodes (6U rack space, approx 2.5 kW under load) to obtain the required performance. We describe the prototyping activities performed at Durham, including estimates of AO system latency and jitter. The design challenges are presented, along with the techniques used to overcome these. The full modular architecture is described, including the system interfaces, control and configuration middleware, telemetry subsystem, and the hard real-time core pipeline. One benefit of our design is the ability to simultaneously test different AO control algorithms, which represents a significant opportunity for automatic optimisation of AO system performance. We discuss this concept, and present an artificial neural network solution for machine learning, which can be used to automatically improve MOSAIC performance with time. Algorithms that can be optimised in this way are discussed, include pixel calibration and processing techniques, wavefront slope measurement routines, wavefront reconstruction techniques and associated parameters, and temporal filtering methods, including vibration control. The hardware design for the real-time control system is presented, including an overview of the network architecture, the interconnections between computational nodes, and the method by which all pixels from all 8 wavefront sensors are processed concurrently.