GOES-16 Advanced Baseline Imager instrument performance monitor

Abstract. Time series generated from NOAA operational satellite sensor temperature, noise, and calibration parameter statistics is a critical science analysis tool to trend instrument performance and detect and resolve on-orbit anomalies. Establishing this capability entails ingesting instrument engineering, housekeeping, and calibration data; performing statistics on them; and then storing and providing the resultant data and/or plots. For instruments with relatively small amounts of input and output data, this is a relatively easy task. For the NOAA Geostationary Operational Environmental Satellite R-Series Advanced Baseline Imager (ABI)—with three times more spectral information, four times the spatial resolution, and more than five times faster temporal coverage than previous GOES—instrument performance monitoring can be extremely complex because of the relatively large data volumes and number of parameters. Also of difficulty is that software and computing architecture needed to build such a system is usually proprietary and not openly documented. In order to fill this gap, we focus on the concept of operations and the results associated with the ABI instrument performance monitor. This monitoring system has proven to be extremely valuable in tracking instrument stability and detecting and performing initial diagnosis of ABI anomalies.


Introduction
After a NOAA operational satellite is launched, monitoring and maintaining instrument data integrity is one of the essential activities related to satellite and instrument command, control, and sustainment. To achieve these goals, it is paramount for each operational satellite instrument to have a supporting instrument performance monitor (IPM). An IPM is a critical science analysis tool to trend instrument performance and detect and resolve on-orbit anomalies. A critical requirement for a NOAA operational satellite IPM is to be able to generate time series from satellite sensor temperature, noise, and calibration parameter statistics by ingesting instrument engineering, housekeeping, and calibration data; performing statistics on them; and then storing and providing the resultant data and/or plots.
For instruments with relatively small amounts of input and output data, creating an IPM is a relatively easy task. For example, an Advanced Microwave Sounding Unit-A instrument onboard an NOAA or EUMETSAT polar-orbiting operational satellite produces about 10 GB of raw IPM data during its 10-year lifetime, and these data contain only about 100 trackable engineering, housekeeping, and calibration parameters. On the other hand, for the NOAA Geostationary Operational Environmental Satellite R-Series (GOES-R) Advanced Baseline Imager (ABI) 1,2 -with three times more spectral information, four times higher spatial resolution, and more than five times faster temporal coverage than previous GOES 3 -instrument performance monitoring can be extremely complex. From the GOES-R ABI raw data stream, the GOES-R Ground Segment (GS) generates level 1b Earth radiance product files and separates instrument calibration data (ICD) files that include the engineering, housekeeping, and calibration data needed to track and trend instrument performance. The ABI ICD is composed of tens of thousands of trackable parameters associated with on-board thermistors, space and internal solar diffuser and blackbody calibration target data acquisitions, and GS generated calibration noise and gain parameters for each of the 7856 operational ABI detectors. The volumes associated with these data can top over 50 TB during an 8.6-year instrument lifetime. Although only about 3.5% of the size of the level 1b radiance product, creating a fast and efficient IPM from these GOES-R ABI ICD requires new paradigms in data handling, storage, and presentation. Commercial systems, such as those built by the GOES-R Ground Segment contract vendor, have concepts of operations that are often proprietary and not openly documented. In order to fill this gap, this paper focuses on the concept of operations (CONOPS) and the results associated with an independently established ABI IPM. This monitoring system has proved to be extremely valuable in tracking instrument stability and detecting and performing initial diagnosis of ABI anomalies.
The GOES-R ABI IPM is a subset of the Integrated Calibration Validation System (ICVS), 4-6 which was developed and is operated at the NOAA National Environmental Satellite Data and Information Service (NESDIS) Center for Satellite Applications and Research (STAR). The primary IPM requirements are to access and input GOES-R ABI ICD-e.g., sensor temperature variables and computed calibration parameters and coefficients and instrument noise valuesimported into the STAR computing environment, and to perform simple statistical analysis on them. The ICD files are obtained with special permission from the NOAA/NESDIS Environmental Satellite Processing and Distribution System (ESPDS) Product Distribution and Access (PDA), but all GOES-R data users can obtain them from the NOAA Comprehensive Large Array-data Stewardship System. 7 Data latency from PDA for engineering ICD is about 3 h and that of the calibration parameters is less than an hour. Once these data are in the local LINUX cluster, the IPM needs to be able to organize and store raw and analyzed ICD data in files with an easily accessible format, and to output time series plots and images of these data for trending.
The baseline version of the IPM was deployed within the STAR information technology architecture before GOES-16 ABI preoperational data began being generated in January 2017, and currently is used to trend both GOES-16 and GOES-17 ABI instrument performance. The GOES-R IPM allows STAR calibration specialists, with local-area computing network access, the ability to comprehensively investigate ABI instrument data quality in near-real time to support anomaly resolution and detailed long-term monitoring. It also permits worldwide satellite operators, science community members, and data users with internet access the ability to browse near-real time, consolidated ABI instrument information and data statistics in a format that fosters product integrity knowledge and anomaly detection. For example, the GOES-R ABI IPM is critical to the GOES-R Calibration Working Group (CWG), which provides ABI calibration technical guidance and recommendations to the GOES-R Program and Projects regarding instrument performance and anomalies. Its members are primarily geographically dispersed working level engineers and scientists at NOAA/NESDIS (including its cooperative institutes and centers), NASA Goddard Space Flight Center, National Institute of Standards and Technology, and Massachusetts Institute of Technology Lincoln Labs. The GOES-R ABI IPM helps to link CWG members with ABI technical information needed to monitor instrument performance and take part in anomaly resolution efforts.
In this paper, Sec. 2 is an overview of the GOES-R ABI IPM architecture, data flow, and output data parameters; Sec. 3 reveals the performance of this valuable tool along with its ability to support ABI anomaly resolution; and Sec. 4 offers a brief summary.

GOES-R ABI IPM Concept of Operations
The GOES-R ABI IPM CONOPS encompasses the basic design, operation, and expected outputs of the monitoring system. In Sec. 2.1, a high-level view of the IPM architecture provides a design and operations summary. Meanwhile, in Sec. 2.2 and its nested sections, the IPM output artifacts used for comprehensive instrument performance analysis and those used to provide insight into instrument temperature and calibration parameters trends are classified and itemized.

GOES-R ABI IPM Architecture
Several design decisions have been made to meet the basic GOES-R ABI IPM requirements to import, analyze, store, and render GOES-R GS ABI ICD data. A schematic of the GOES-R IPM architecture can be found in Fig. 1. Starting at this figure's left-hand side, the GOES-R GS ICD files-containing sensor temperature variables, calibration parameters and coefficients, and computed instrument noise values-are ingested by STAR in near-real time from the NOAA ESPDS PDA utility, and stored for a limited time in the STAR Central Data Repository (SCDR).
Multiple times per day, the GOES-R IPM software will search for the existence of new input data sets within SCDR and performs necessary data manipulation and statistical analyses to update instrument performance long-term-i.e., monthly to mission life-time series stored into GOES-R ICVS Dataset Storage (GRIDS). The GRIDS is a file system carefully designed with different data uses in mind. For example, the GRIDS files are broken down into those that contain long-term time series of native time resolution data, and files containing only the most recent native time resolution data or time-degraded statistics of long-term time series. The GRIDS files encompassing long-term time series of native time resolution data provide detailed deep-dive understanding that is critical for anomaly exploration by the STAR calibration specialists. Meanwhile, the GRIDS files that include time-degraded statistics are relatively small-sized data sets that can be used to generate plots and images as follows. Once the GRIDS files have been updated, the software then performs instrument performance near-and long-term time series analysis and graphics generation for the purpose of manifesting web-ready GOES-R IPM plots and images. These graphics can be served via the GOES Calibration website 8 to all users in order to foster high-level instrument and product quality situational awareness.
A more detailed illustration of the GOES-R IPM data processing flow can be found in Fig. 2, to support visualization of the GOES-R IPM processing. The top half of Fig. 2 shows that the IPM pulls new data accumulated between the end time stamp of the most recently processed ICD file and the current time, from the SCDR; performs necessary arithmetic manipulations to these data; and then updates the appropriate GRIDS long-term time series. In these initial steps, up to one full day of new data at a time can be processed and added to the GRIDS files, which after full processing contain time series that range from satellite beginning of life to the end time stamp of the ICD file nearest to the current time. STAR calibration specialists can use the long-term time series of native time resolution data that are produced during this process and stored in GRIDS to perform deep-dive anomaly exploration studies. Meanwhile, the lower half of Fig. 2 illustrates that data are imported from the updated GRIDS data files, analyzed with standard time series analysis, and then exported to time series and imagery that can be sent to the web.

GOES-R ABI IPM Output Parameters
The following sections itemize the GOES-R IPM output artifacts separately for the GRIDS file parameters needed for deep-dive analysis, and the parameters made available to users via the STAR Web Server.

GRIDS file parameters
As mentioned above, GRIDS is the name given to the integrated NetCDF file system that accumulates time series artifacts from the individual GOES-R ICD files. GRIDS has two types of files: files that contain long-term time series data that are resolved as possible in the time domain for the purpose of deep-dive instrument performance studies and anomaly resolution by STAR calibration specialists; and files that contain short-term full-time-resolution data, as well as mission-life statistical data with 1-h and 1-day time resolution, which are used to create web artifacts that can be rendered quickly.
GOES-R instrument deep-dive instrument performance and anomaly resolution will often entail analysis that requires "no-data-left-behind." This means when possible all the ICD must be available at their native output time resolution for inspection and analysis by STAR calibration specialists. The structural name of these files, their associated primary data contents, time period stored, and time resolution are given in Table 1.
Displaying time series on the internet is limited by the number of computer monitor pixels in the horizontal and vertical screen directions. For most computer monitors, the resolution is no more than 1920 × 1080 pixels. Thus the ability for a computer monitor to render time series with thousands or more data points become a challenge. In addition, the speed in which time series are rendered can slow dramatically as the number of data points increases. For this reason, longer time series are created with parameter statistics generated at varying time intervals. 9 The contents of the GRIDS files that hold these data are given in Table 2.   Critical automated software tools using the GRIDS data files have been developed by STAR calibration specialist to capture ABI anomalies in near-real time. These anomaly detection tools can determine out-of-bounds parameter values in native time resolution data taken in the mostrecent few hours, using a data baseline developed from the 1-h and 1-day statistical data over a longer time span-e.g., the previous 10 days. These alert tools can capture an error in any one of the 7856 ABI detectors and report it to a calibration specialist in an e-mail. Once an alert analysis has been completed, it can be stored for later inspection or analysis. Without these tools, detection of ABI anomalies would require extremely cumbersome daily manual inspection and analysis.

Web server parameters
The web interface for the GOES-R ICVS can be found in Ref. 10. Images are served to the web in a dynamic way with the help of comma separated variable files that include fields for the select group, image file path, filename stub, label, and index. These fields help to make the required links between an IPM monitoring parameter chosen by the user in a web page and IPM graphics artifacts stored in STAR web server memory that are to be rendered on the screen. The types of parameters that are currently available are listed in Table 2. Meanwhile, Fig. 3 provides some insight into the nature of the GOES Calibration web site instrument monitoring navigation and content layout.
In Fig. 3, the GOES-16 ABI Cal/Val Instrument Monitoring tab in the web page's left-side scan column consists of links to visible-near infrared (VNIR) and infrared (IR) band operational calibration, solar calibration evaluation, and engineering telemetry statistics artifacts. This particular image shows ABI Mode-6 (Ref. 2) Band 1 (0.47 μm) detector-by-detector spaceview counts within the context of the VNIR band operational calibration page. This page includes three drop down boxes that allow web site users to choose a parameter, an associated subparameter, and a date. The upper-right corner of the web page also is equipped with options that allow the user to animate all images for a given parameter, animate all parameters, and find a given parameter in the page. This image also shows the ability to navigate through the web site using breadcrumb navigations on top.
In Fig. 4, examples of GOES-16 ABI Mode-6 IR band operational calibration noise equivalent delta temperature (NEDT) parameters are shown with two subplots. The first subplot is of ABI Band 7 (3.9 μm) maximum, minimum, and mean NEDT at time scales of 2 days, 10 days, and 30 days, as well as for 1-year and mission life. A second subplot, similar to the other, represents NEDT for all IR channels over the past 30 days. The IR band operational calibration link also provides the web user glimpses in the trend of blackbody and spacelook counts, instrument gain, and provides insight into the most prevalent detector anomalies. Note that plots showing a detector-by-detector perspective of the IR data are available, but plots similar in format to the ones shown in the previous figure for the VNIR channels are also available.
Although the gain of the ABI IR channels needs to be updated every 5, 10, or 15 min at the beginning of each Mode-4, Mode-6, and Mode-3 timeline, respectively, the VNIR channel gain changes much more slowly. Shortly after launch of an ABI instrument, solar calibration and associated spacelook target measurements are taken every few days and are used to update the gain of the VNIR channels to maintain data integrity. As the mission continues, the frequency of these solar calibration events decreases to about once every 3 months or less. This important activity is captured within the GOES-R Calibration web site in the "Solar Cal Evaluation" web page. In Fig. 5, the ratio of the on-orbit GOES-R ABI VNIR band gain relative to values computed during prelaunch testing are shown for all six VNIR channels. The solar calibration evaluation web page also includes plots and/or tables related to detector-by-detector signal-to-noise ratio (SNR), solar calibration and spacelook target count measurements, solar calibration time, and solar geometry relative to the solar diffuser aparatus and the diffuser.
Instrument engineering telemetry-e.g., instrument subassembly and component platinum resistance thermometer (PRT) temperatures-can be key in understanding calibration performance. All of the ABI engineering telemetry sensor PRT temperature trends can be found in the "Engineering Telemetry Stats" link in the instrument monitoring section of the web site. In Fig. 6, the GOES-16 ABI blackbody PRT temperature trends are shown at time scales of 2 days at full 1-s time resolution, and at 10 days, 30 days, and mission life at 1-h time resolution. As can be seen from these examples provided in this section, the GOES-R Calibration web site offers an extensive catalogue of plots and data that provide situational awareness into GOES-R ABI calibration for all GOES-R ABI users.

GOES-16 ABI IPM Results
For GOES-16 ABI, there were a series of tests that the IPM needed to pass as part of product validation. The success criteria associated with these tests that needed to be met are the following.
• ABI ICD fidelity criteria. Raw telemetry and calibration data represented in plots, images, and tables are correct with respect to their ICD source. • ABI IPM software execution expediency criteria. Raw telemetry and calibration data can be processed faster than the rate that they are acquired by ABI, processed by the GOES-R GS, and delivered to STAR through ESPDS PDA. • ABI IPM GRIDS and output artifact availability criteria. The instrument performance monitoring data set represents at least 95% of available telemetry and calibration data available for processing.
In the following sections, the results of this test are evaluated with respect to these success criteria. Finally, some examples of how the IPM has been used to detect and correct ABI instrument anomalies are given.

ABI Instrument Calibration Data Fidelity Criteria
From observed blackbody, space view, and solar diffuser samples, it is determined that the minimum, maximum, and mean statistics computed from these samples are in agreement to those delivered by the ABI GS to within machine precision. At present though, the fidelity of some of the ABI ICD cannot be validated. The reason for this is that, at the time of the writing of this paper, no credible engineering telemetry data have been manifested in the ICD files, which are needed to reproduce gain measurements in the IR channels. This test produces only limited success, and important calibration data-engineering telemetry and IR gain-cannot be validated.

ABI IPM Software Execution Expediency Criteria
The amount of time it takes for the ABI IPM software to process the ABI ICD in order to establish both the GRIDS, as well as all of the portable network graphics image artifacts, is a small fraction of the actual data accumulation rate-i.e., the rate that data are acquired by ABI, processed by the GOES-R GS, and delivered to STAR through PDA. Recent runs of the software shown in Table 3 reveal that the longest runtime, 210 s to process 4 h of ABI Channel 2 ICD, is only 1.46% of the data accumulation rate.

ABI IPM GRIDS and Output Artifact Availability Criteria
In Table 4, for each GRIDS file type, the following accessibility and availability statistics are given: percentage of GOES-R GS ABI ICD that are accessible to the IPM the first 6 months after   GOES-16 ABI L1b provisional validation maturity declaration (June 1 and December 1, 2017); and the percentage of these accessible data that are made available in the GRIDS. Similarly, the GRIDS files that hold the reduced-size time-interval statistical data are also given in Table 4. There are four main reasons why there is less than 100% GOES-16 ABI ICD accessible. First, before July 24, 2017, Channel 2 ICD was generated by the GOES-R GS without a valid data chunking scheme, which prevents file sizes from becoming extraordinarily large. Until this date, each five-path suite of ICD files for Channel 2 could be as large as 5 GB, which made them too large to move and process in a reasonable amount of time. Second, there were data outages up to 10 days each due to disruption of ICD generation. This happened twice when the GS increased the space view counts threshold for lunar intrusion. The ground segment is currently creating a standard operating procedure to make sure that the ICD generation service is restarted after making such changes. Third, there are instances where the five data paths of the Channel 2 ICD have different numbers of data records. Such errors make it very difficult to stitch the data together for processing in a reasonable time. Finally, during GS DO.05/DO.06 build testing at the end of July/November 2017, the instrument calibration engineering files were not filled with data for about 10 days during each outage event. As the GS is now familiar with these issues, any recurrence should not be as impactful.

ABI Instrument Events Detected by the IPM
Between GOES-16 ABI provisional maturity declaration on June 1, 2017, and May 10, 2018, the GOES-R ICVS IPM detected 10 instrument events. In Table 5, a list of nine of these calibration events is given.
The nine calibration events listed in Table 5 do not have to be described individually, as some of the events can be grouped according to their common root cause. These groups are as follows: unphysical 10% gain jump, unexpected gain variations between solar calibration events, unjustified change in solar irradiance values, and spacelook mean counts latch-ups. Most of the above are caused by ground segment look-up table (LUT) anomalies, whereas the others are initiated by detector anomalies that trigger a condition in ABI L1b processing that creates anomalous gain values. As part of the GOES-R ICVS web site, there is a list of all postlaunch anomaly events for each ABI instrument, which can be found in Ref. 11. Meanwhile, the events listed in Table 5 are described below.

10% VNIR gain jump
GOES-16 ABI VNIR bands had several 10% gain jumps events since it began taking VNIR imagery in January 2017. These gain jumps impacted the earth radiance and user products. Each gain jump event typically occurred simultaneously in all VNIR bands but also was found to occur in individual bands for some events. Generally, the 10% gain jump has been caused by the GOES-R GS implementing solar calibration coefficients computed using an integration time factor of 9 instead of 10. It is important to keep a clear record of timing of these 10% gains jumps. For this purpose, the GOES-R CWG utilizes solar calibration event trending and VNIR band calibration monitoring results. Figure 7 indicates the five main periods of 10% gain jumps from calibration monitoring performance since the start of 2017, with detailed timings provided in Table 6. Note that Band 5 (1.6 μm) had a different earlier start time in Case 4 and a delayed ending time in Case 5. However, the exact timing might not be reflected due to lapses in the ICD files. All of dates were extracted based on monitoring and trending.
Although five periods of VNIR 10% gain jump were monitored, Case 3, 4, and 5 from Table 6 are associated with periods where the correct solar calibration coefficients were computed by the GS, but there were errors in the LUT. Figure 8 indicates the solar calibration trending for each band. Two periods (February 11-17, 2017, and March 19 to April 20, 2017) had 10% gain jumps that were clearly identified in bands 1 to 3 (0.47, 0.64, and 0.86 μm). For Bands 4 and 6 (1.37, 1.6, and 2.2 μm), the GS did not record the gain in the solar calibration ICD files for February 11-17, 2017, 10% jump, thus no red points are plotted there. During April, June, and July 2018, VNIR channel gain changes occurred that were not necessarily associated with solar calibration events. In these instances, gain changes happened as frequently as once per ABI timeline and were found to be due to GS "Software Build" testing requiring an operational processing string and a "shadow operations" processing string to run simultaneously. These two software strings did not necessary hold the same ABI L1b processing input parameter LUTs and their outputs became intertwined. An example of this is shown in Fig. 9, where GOES-16 ABI Band 6 gain variations are plotted for the end of June to the beginning of July 2017. These frequent erroneous gain changes did not occur during later ground segment testing in October and November 2017.

Unjustified change in ABI solar irradiance values
During GOES-16 ABI postlaunch testing, VNIR in-band total solar irradiance (Esun) values reported in the ABI L1b files were found not to be the same as provided by the ABI vendor before launch. There also was an unjustified change of these values during this time (see Table 7), and the root cause of these unjustified changes is not known. These values were updated properly with DO.06.00 in June 2017.

Spacelook mean counts latch-ups
The ABI L1b algorithm screens the spacelook mean count values for increases that may be caused by the moon entering the spacelook field of view. When the spacelook mean counts surpass a threshold, they are held constant, or latched up, to the last "good" spacelook mean counts value. This latch-up ends when the spacelook mean counts fall below the threshold expected for moon-in-spaceview events.
There have been several occurrences where ABI spacelook and internal calibration target (ICT) detector mean counts spontaneously shift together to larger values for individual detectors. Thus these spacelook counts mean value changes are not due to moon-in-spaceview events. On the other hand, if the counts increase exceeds the thresholds defined for moon-in-spaceview events, then the spacelook counts mean is similarly latched up to the last good spaceview. The problem is that this counts "level-shift" condition does not always go away by itself, and the counts may not ever become small enough in value to reset the moon-in-spaceview flag.     Meanwhile, the ICT mean counts continue to fluctuate with the diurnal cycle, which causes the computed gain to erroneously vary with the diurnal cycle. This phenomenon can be visualized with the aid of Figs. 10-12. These figures shows how latched up spacelook mean counts values can lead to gain irregularities. It is these gain irregularities that in turn lead to stripes in ABI L1b product. The algorithm to eliminate lunar intrusion level changes is limiting the underlying algorithm that corrects for real detector level changes through frequent space look removal. Although a partial mitigation is being investigated with WR5730 for the short term, a longer term resolution is to be implemented.

Summary
This paper has provided insight into the CONOPS and results of the GOES-R ABI instrument performance monitoring system built as part of the STAR ICVS. This system is capable of inputting, analyzing, storing, and providing outputs relating to tens of thousands of trackable parameters that have a total data volume that can top over 50 TB over an 8.6-year instrument lifetime. Creating such a complex system requires significant thought and resources, but documented ideas about building such a system are usually commercial, and thus proprietary. This paper coarsely fulfills this need to document the high-level architecture and data and analysis flow associated with such a system.
During postlaunch product test checkout of the GOES-16 ABI IPM, it was found that the IPM partially met its success criteria. Success has been observed in the ability to process all available ICD from the GOES-R Ground Segment via ESPDS PDA, and in software execution times that are at least one hundred times faster than the data are created. This paper also was able to highlight several examples of how this tool has transformed a difficult job of tracking and trending ABI instrument performance and anomalies into one that supports GOES-R ABI instrument and calibration integrity in near-real time during postlaunch checkout and operations.