6.1.

Control: Telescope Manager Architecture

The SKA chose Tango (Tango is developed, managed, and maintained by Tango Controls⁶), an open-source SCADA-like toolkit, to form a distributed framework for the entire telescope’s monitoring and control system, including the TM itself. Using Tango, the implementation of software interfaces and standard functions such as logging and error handling are greatly simplified.

Figure 6 shows the structure of the Tango-based TM, as tailored to the SKA. Tango, which by design is hierarchical, leaves much of the control and monitoring complexity to the downstream Telescope sub-systems. The central concept behind Tango is the Tango Device, a software object that models “real-world” equipment and provides a standard interface for access. Devices can either be located on the same computer or distributed across computers and linked by a network. One of the reasons for the hierarchical architecture, besides distributing the complexity of a large system, is reducing information flow back and forth, as would occur in a centralized approach. Nevertheless, information flow is not blocked; Telescope Monitor and Control (TMC) devices can access any single Tango device without respecting the hierarchy. Tango also provides a framework for logging, archiving, alarm handling, and operational monitoring.

Fig. 6

An illustration of the typical TANGO LMC and device hierarchy. TM interfaces with multiple devices with a dynamically variable hierarchy.

The SKA Control Systems Guidelines tailored the application of Tango to establish a hierarchical control structure in which a sub-system has a designated Local Monitor and Control (LMC) that adheres to the Tango framework to provide a harmonized approach to control and monitoring. Each controlled component within the telescope interfaces to its LMC with a Tango Device that interfaces on one side to the Tango framework and on the other side to the low-level control functions of that system.

The resultant monitoring and control architecture for SKA1-Mid is shown in Fig. 7, showing both static devices associated with real hardware, such as a dish, and dynamic ones, associated with a virtual construct, such as a sub-array of dishes.

Fig. 7

The monitoring and control architecture of SKA1-Mid, using the Tango framework. The grey lines represent general communication between devices, and the red, blue, and green lines show communication between dynamic devices created to establish three sub-arrays for a specific observation.

Proposal preparation, approval, planning, and scheduling are done through a family of Observatory Science Operations tools, which create detailed scripts for each observation in a database. The Observation Scheduling Tool (OST) is used to establish the macro observing schedule while the Observation Execution Tool (OET), shown at the top of Fig. 7, accesses this database and, under guidance of the Telescope Operator, sets up and executes the observation. The scripts are executed by the appropriate Tango Devices in the various hierarchical layers. The top co-ordination is in the Telescope Monitoring and Control (TMC) layer, which in turn communicates with the devices that form part of the other telescope systems, forming an integrated controls architecture that spans the divide between telescope sub-systems. The multiple, dynamic instances of Tango devices provide great flexibility and modularity, a key aspect of the SKA design.

SKA1-Mid is required to do commensal observing in which available resources are used concurrently for different observing projects, depending on their needs. The control architecture facilitates this through the concept of multiple “sub-arrays,” to which resources are allocated and which can observe simultaneously. A Sub-array Node controls each sub-array’s resources with the overall orchestration by the Central Node, as shown in Fig. 7.

6.2.

Correlator-Beamformer Architecture

The Correlator–Beamformer (CBF) section of the signal chain shown in Fig. 5 carries out digital signal processing (DSP) for multiple sub-arrays to support the following:

Figure 8 shows these outputs and their context in the overall architecture. The CBF system must support very wide bandwidths and a relatively large correlation matrix of output visibilities. Processing this much digital data in serial fashion is not possible with current technology. The architecture of the CBF is designed to divide the wide bandwidths into 27 narrower, 200 MHz “frequency slices,” which are processed in separate DSP engines (Fig. 9). Each wide band is delay-corrected and is ‘sliced’ by the Very Coarse Channelizer (VCC), and the sliced data are sent on to Frequency Slice Processors (FSPs), which can carry out any of the processing functions (listed above) on a particular slice. This approach can, for example, enable the real-time processing of up to 5 GHz of RF bandwidth from 197 dishes, yielding $\sim 5\times {10}^9$ channelized complex-visibility products in integration times as short as 0.14 s.

Fig. 8

A context diagram for the CBF system. The CBF, PSS, PST, and their LMC system are collectively called the central signal processor (CSP). The blue labels show the outputs.

Fig. 9

(a) A simplified version of the frequency slice CBF architecture. (b) An illustration of how the RF band is split into slices, where each slice is processed independently of the others.

Figure 10 shows the flow of time-stamps and synchronization of data samples for the CBF in the context of the entire SKA1-Mid system. Throughout the system, time-stamps are carried by 1-PPS marker signals. These pulses are effectively labeled with UTC time once per second (i.e., time-stamped via an Ethernet connection that ensures that there is no ambiguity between the 1-PPS pulses).

Fig 10

The flow of time signals and synchronization of data-samples through the SKA1-Mid CBF in the context of the SKA1-Mid system. Although only a single dish is shown, each dish provides sampled data to the CBF in the same way.

A detailed explanation of the versions of 1-PPS shown in Fig. 10 are provided in Ref. 1. As a result of a simple transfer, the time-stamps provided at each dish (WR-1PPS) are transferred to the A-1PPS, which are embedded in the data stream, effectively labeling each sample with UTC. There is a small residual error, which is ultimately calibrated out by observing sources on the sky. A delay model is provided to the CBF system, which provides the sum of the atmosphere/ionosphere, geometrical delay, dish optical delay, and analog RF path. Variations in the electronic delay from the samplers to the CBF are “soaked up” in the wideband input buffer (WIB). Using the time stamps, the data streams from all of the dishes are re-gridded onto sample streams that run at a common rate for all dishes (re-sampler/delay tracker). The delay is compensated for in the same interpolation operation. The common sample rate is slightly higher than any of the input sample streams so that there is never an overflow of data. Smaller variations in arrival times are soaked up in the Synch Buffer. At this point, the samples are completely aligned and can be correlated or beam-formed.

6.3.

Architecture of Pulsar and Time-Domain Processing

As shown in Fig. 5, SKA1-Mid contains two specialized processors, one for searching for pulsars (PSS) and the other for making precise measurements of their times of arrival (PST).

The scientific goal of PSS is to enable the search of the entire sky for pulsars and time-domain transients visible from the site. Figure 11 shows a typical pulsar signature and the methods for searching. Searching involves finding the unknown dispersion measure (measure of curvature) as well as the unknown pulse period. Only when search beam data are de-dispersed and stacked over the frequency dimension, is there sufficient signal-to-noise to enable searching the time series for pulses. Search efficiency is dependent on being able to simultaneously search as large an area of sky as possible. Hence the CBF provides 1500 array-beams on the sky, which can be steered independently within the beam of the individual dishes. Typically, only an optimized fraction of the central part of the array is used so that each of the array-beams is not too small. The PSS system is implemented on a large system of GPU processing units distributed over $\sim 33$ racks.

Fig. 11

Top: The time-frequency signature of a dispersed pulsar pulse before (left) and after (right) de-dispersion. Bottom: A simplified view of the pipeline of signal processing used in searching for pulsars. This is carried out in parallel for all beams.

Transients (single pulse, dispersed events) are detected when a single pulse is strong enough to exceed a pre-determined signal-to-noise ratio (SNR). When this happens, a trigger-signal is sent to the CBF to freeze the contents of a buffer to capture the signal from each dish. The captured time series is processed off-line.

In PSS, the algorithms within the boxes in Fig. 11 are optimized for speed, not maximum precision. Pulsar timing (PST) utilizes a similar underlying method to that shown in Fig. 11, but since the observations are of known pulsars, with times of arrival that are tracked over years (at least a decade), the algorithms, particularly de-dispersion, are optimized for accuracy, limited only by the SNR. Only 16 array beams are required for PST, but much wider bandwidth is accessible. Also, because beam-area is not important, more dishes can be used.

De-dispersion is essentially a digital filtering operation. It can be done either incoherently on a digitized sample stream of squared “voltages” in which phase information is lost or coherently on a complex sample stream of digitized voltages in which phase information is retained. PSS uses the former method, which is much faster, whereas PST uses primarily the latter, although it uses incoherent methods as well.

6.4.

Science Data Processor Architecture

The SDP is a complex software suite that provides various pipe-line processes that can be executed to calibrate the telescope and analyze the data based on the defined observational requirements. Figure 12 shows the architecture, the flow of data, and the functions performed by the SDPs. The data shown in the top box originates in the CSP system, co-located with the SDP system at the SPC in Cape Town, and is transmitted to SDP, an HPC. Because SDP is configurable and must process data at the same average rate at which it is produced by CSP, it is under the overall control of the TM.

Fig. 12

SDP data flow, architecture, and functions.

Most data are ingested at a high rate, stored temporarily in a buffer, and passed to parallel batch mode processors. However, some data, such as pointing calibration data for dishes, ionospheric calibrations, and responses to transient events, must be processed in a time-critical manner and fed back to the TM. As shown at the bottom of Fig. 12, batch-mode data products are provided to the various SRCs.

SDP receives two types of observational data from CSP. These are visibility data (correlator outputs), received as a continuous flow to be imaged, and non-imaging data (transient buffer, pulsar and transient search candidates, and pulsar timing data) received as discrete chunks. In cases in which there are commensal observations, the same data may be processed as both imaging and non-imaging data.

In addition to science data products, the SDP computes calibration data, generates metadata, generates alerts, and maintains Local/Global Sky Models. Additional information is generated to track and assess the efficacy, throughput, and quality of the data production.

The SDP interfaces to the SKA control system and time-critical processing is directly scheduled by TM. Production of non-time-critical data products is performed in a batch-oriented processing mode: the overall science scheduling of the telescopes is linked to the available compute and data-storage resources of the SDP, so the overall throughput of the processing does not result in blocking of observations.

Figure 13 shows the context of the SDP in terms of its data interfaces.

Fig. 13

Diagram depicting the context of the SDP in terms of the information and data communicated to systems.

The SDP challenge has aspects that, when considered together, make it unique among comparable systems in astronomy.

7.1.

Realization of the SKA Timescale and Time Distribution Systems

The timescale is synchronized through GNSS satellites using well established methods and protocols. This puts the SKA telescopes on a footing similar to other timescale realizations such as UTC (NPL) and UTC(PTB). BIPM calculates UTC in retrospect from these timescale realizations, accounting for the uncertainties of each UTC (k), and publishes corrections with a delay of 30-50 days in what are known as the Circular-T publications.

Figure 15 shows the components of the timescale:

Fig. 15

The three hydrogen masers making up the timescale reference are steered to align with the UTC time received from BIPM to provide an accurate local UTC time instance that forms the basis of SKA1 Mid timing and synchronization.

Fig. 16

A simplified diagram of the reference frequency round-trip distribution system without the clean-up oscillator. The outlined sub-diagram is a concept diagram showing mapping into the Michaelson interferometer.

The outputs of the timescale sub-system are as follows:

The common time (1-PPS) signals, often referred to as a heartbeat signal, are distributed throughout the telescope system. The rising edge of the 1-PPS signal is accurately aligned to that generated by the SKA timescale [UTC(SKA)]. A so-called White Rabbit (WR) system is used to distribute this pulse to all of the dishes in a way that maintains its accuracy despite the $\sim 100\mathrm{km}$ distance to the farthest dish. WR provides an open-source protocol based on a hardware rendition of the Precision Time Protocol (PTP, IEEE1588v2). It utilizes a feedback system to maintain alignment of the remote delivery points with the 1-PPS output of the timescale sub-system.

The timescale is expected to have an uncertainty of less than 4.8 ns (1-sigma), and the accuracy of the 1-PPS distribution is better than 1.5 ns to the farthest dishes.

7.2.

Frequency Distribution for Array Synchronization

The design challenge here is to distribute a sample clock frequency to the ADCs in each dish that is locked to the reference frequency produced by the timescale, with a stability of better than tens of femto-seconds over 60 s.

Although the preferred distribution medium is optical fiber, it is not free of noise and its effective electrical length varies with environmental changes (temperature/vibration). This adversely impacts the output signal quality (its stability and accuracy) as it is conveyed up to $\sim 100\mathrm{km}$ from the CPF. The distribution path is a combination of buried fiber, fiber placed on overhead lines (i.e., strung between poles), and fiber passing through the cable wrap in each of the dishes. The distribution system must handle the temperature and mechanical stresses on the fiber over paths with these characteristics.

SKA1-Mid employs a feed-back controlled method, often called a round-trip compensation system, which actively compensates for delays along the reference frequency distribution path.⁸ Any frequency transfer technique is ultimately limited by the fraction of the noise spectrum that can be suppressed by the feedback. Random noise processes or disturbances that occur at timescales faster than the round-trip light travel time will have de-cohered the output before this interval has occurred. The servo loop cannot sense these disturbances and therefore cannot correct for them, so a ‘clean-up’ oscillator is employed to filter out the high-frequency noise. This is implemented using a high-quality, low-noise crystal oscillator phase-locked to the output of the round-trip system.

A nominal 3.96-GHz sampling signal is synthesized from the SKA1-Mid timescale 10-MHz reference by the FRQ system and distributed over optical fiber to the focal platforms of each of the 197 dishes, where it is distributed within the receiver/digitizer box to each ADC clock input. This is sent directly to the focal platforms of dishes in the core, but for dishes that are farther away, bidirectional EDFA optical amplifiers are inserted in the path. These are located either in the pedestals of some dishes or in special repeater shelters.

Figure 16 shows a conceptual view of the round-trip feedback method for one dish. The detailed block diagram and description are provided in Ref. 9. The system comprises a Michaelson interferometer (see top sub-diagram) in which the top arm acts as a fixed reference length. The acousto-optical modulator (AOM) consists of a laser source that feeds the two arms of a Mach–Zehnder Interferometer (MZI) (not shown in detail). In one arm, the static microwave reference frequency (double the sample clock frequency, 7.92 GHz) which is to be delivered to the remote end, is applied by shifting the optical signal from the laser source. When the two arms are recombined at the output of the MZI, this results in two optical signals on a single fiber with the microwave-frequency separation. This signal is transmitted over the optical fiber link to the remote telescope site where the two optical signals mix (beat) in a photodetector, thus recovering the original reference microwave signal.

A fraction of the two transmitted optical signals is reflected back to the central transmitter site. A Faraday mirror is used to rotate the plane of polarization by 90 deg so that it can travel on the same fiber without interacting with the transmitted signals. The reflected signals are mixed with a copy of the transmitted optical signals in a photodetector at the source, yielding several mixing products, some of which are at the microwave reference frequency and contain twice the fluctuating delay in the long arm. Two of these signals are then mixed with a copy of the microwave signal to produce low frequency RF signals. These are further processed to produce an error signal, an offset frequency that encodes the fluctuations of the link. Applying this as the drive signal of the AOM closes the servo loop and effectively cancels the link error for the remote site. In the full-blown system description, given in Ref. 9, it is shown that several sources of phase noise are also canceled. The reference frequency is 7.92 GHz. This frequency is replicated at the output, whereupon it is divided by two (not shown in Fig. 16) to produce the require 3.96-GHz sample clock reference.