Starshade Exoplanet Data Challenge

Starshade in formation flight with a space telescope is a rapidly maturing technology that would enable imaging and spectral characterization of small planets orbiting nearby stars in the not-too-distant future. While performance models of the starshade-assisted exoplanet imaging have been developed and used to design future missions, their results have not been verified from the analyses of synthetic images. Following a rich history of using community data challenges to evaluate image-processing capabilities in astronomy and exoplanet fields, the Starshade Technology Development to TRL5 (S5), a focused technology development activity managed by the NASA Exoplanet Exploration Program, is organizing and implementing a starshade exoplanet data challenge. The purpose of the data challenge is to validate the flow down of requirements from science to key instrument performance parameters and to quantify the required accuracy of noisy background calibration with synthetic images. This data challenge distinguishes itself from past efforts in the exoplanet field in that (1) it focuses on the detection and spectral characterization of small planets in the habitable zones of nearby stars, and (2) it develops synthetic images that simultaneously include multiple background noise terms -- some specific to starshade observations -- including residual starlight, solar glint, exozodiacal light, detector noise, as well as variability resulting from starshade's motion and telescope jitter. In this paper, we provide an overview of the design and rationale of the data challenge. Working with data challenge participants, we expect to achieve improved understanding of the noise budget and background calibration in starshade-assisted exoplanet observations in the context of both Starshade Rendezvous with Roman and HabEx.


Introduction
Data challenges have advanced the planning and development of major astronomy facilities both on the ground and in space. Science communities participate in the data challenges to analyze simulated data and gain insight into the detection capabilities of the instrument; in turn, instrument designers learn the precision and noise level needed to reveal the objects and phenomena that are looked for. For example, a series of image analysis challenges have been carried out to develop and test methods to measure weak gravitational lensing from small distortion of galaxies's shapes (e.g., the GRavitational lEnsing Accuracy Testing (GREAT) 3 1 ). These community exercises have been 1 arXiv:2103.09359v1 [astro-ph.EP] 16 Mar 2021 instrumental in consolidating the science plans for space missions like Euclid. Another prominent example is the data challenges organized by the teams of LIGO-Virgo 2 and LISA, 2 which helped quantify the measurement precision needed to detect gravitational waves and inform the development of these experiments. Data challenges have also advanced many exoplanet projects.
For example, a radial velocity fitting challenge was conducted to find the most efficient ways to extract planetary signals embedded in stellar noises. 3,4 For another example, a carefully planned data challenge has helped resolve discrepancies among groups in their data reduction and analysis approaches in exoplanet transit observations with Spitzer. 5 More recently, community data challenges have been carried out to derive capabilities and inform instrument designs for the exoplanet science with Roman Space Telescope. 6,7 Past efforts include a challenge to efficiently identify and analyze exoplanetary microlensing events from large datasets, 8 and the Roman Exoplanet Imaging Data Challenge. 9,10 The latter, probably the first community data challenge for space-based exoplanet imaging 1 , includes efforts to validate models of planetary reflected-light spectra at Roman's Coronagraph Instrument (CGI)'s wavelengths, to detect planets from simulated Roman images, and to determine planetary orbits from multi-epoch observations. The Roman Exoplanet Imaging Data Challenge has also explored enhanced planet detection and orbit determination capability with a starshade rendezvous at a late phase of the mission, 11 while still focusing on detecting Jupiter-sized planets. At the time of writing, the final results of the Roman Exoplanet Imaging Data Challenge have not been published; but the atmospheric modeling effort (as Phase I of that effort) succeeded in establishing the ability to retrieve key atmospheric parameters from simulated planetary spectra. 9 Together with the development of the concepts of Starshade Rendezvous with Roman 11 and Habitable Exoplanet Observatory (HabEx), 12 both involving a starshade, NASA's Exoplanet Exploration Program (ExEP) is executing the Starshade Technology Development Activity to TRL5 (S5) to rapidly mature the technology and close gaps in optical performance, formation flying, and mechanical precision and stability. Together with S5, ExEP has chartered a Science and Industry Partnership (SIP) to engage the broader science and technology communities during the execution of the S5 activity. A key recommendation that emerged from SIP meetings and discussions is to produce "a flow down of requirements from science to key performance parameters based on synthetic images (rather than scaling formulas only)" and "a plan for the starshade data challenge." Responding to the community recommendation, S5 is now organizing and implementing a Starshade Exoplanet Data Challenge.
The Starshade Exoplanet Data Challenge seeks to verify and improve the exoplanet yield estimates 13-16 by using synthetic images that realistically capture instrumental effects due to the starshade and the telescope. With the completion of most of S5's technology milestones on instrument contrast, 17, 18 solar glint, 19 and formation flying, 20 we can now simultaneously include in the images multiple sources of background and noise including residual starlight, solar glint, exozodiacal light, detector noise, as well as variability resulting from starshade's motion in formation flight and telescope's jitter. Many of these terms are specific to starshade observations; while some of them may be included in past exoplanet yield estimates, 13-16 the interplay of these terms of background and their noises can only be revealed and evaluated with the analyses of synthetic images.
A key science question that the Starshade Exoplanet Data Challenge is designed to answer is to what extent the background can be calibrated in the context of starshade-assisted exoplanet imaging. If the background is removed to its photon-noise limit, Starshade Rendezvous with Roman could provide nearly photon-limited spectroscopy of temperate and Earth-sized planets of F, G, and K stars < 4 parsecs away, and HabEx could extend this capability to many more stars within 8 parsecs. 16 To achieve these capabilities, the flux of exozodiacal light within the planet's point spread function often needs to be calibrated to a precision better than 1% and the solar glint better than 5%. 16 The challenges for photon-limited background calibration may come from the fact that the solar glint varies with the solar angle and the starshade's position and orientation 19 and that an exoplanetary dust disk is likely inclined and may have structures created by dynamical interactions with embedded planets. 21,22 Also, the expected use of slit-prism spectroscopy by Roman CGI may create complexity in spectral extraction together with the background. The Starshade Exoplanet Data Challenge will thus provide the opportunity to quantify the accuracy and precision of noisy background calibration for detection and spectral characterization of small exoplanets.
We expect the outcomes of the data challenge to help NASA identify and prioritize the areas of future technology development. As we assess the abilities to extract planets, dust structures, and their spectra from images that include varying levels of instrumental effects, we will improve our understanding of how critical the instrument performance parameters are. For example, we learn from signal-to-noise ratio calculations that an instrumental contrast of 10 −10 is likely not needed for many science observations with Starshade Rendezvous, while suppressing solar glint and other stray light sources is of paramount importance. 16 The data challenge will tell us, with fidelity, how much performance would be lost if the contrast were 10 −9 and the brightness of solar glint were a few times higher than the current best estimate (CBE). To summarize, the Starshade Exoplanet Data Challenge is designed to validate the flow down of requirements from science to key performance parameters, quantify the required accuracy and precision of noisy background calibration, and prepare the science community for analyzing starshade exoplanet observations.   S5 will simulate the images for the data challenge. The images will be generated with the Starshade Imaging Simulation Toolkit for Exoplanet Reconnaissance (SISTER, 23 http://sister. caltech.edu), which takes into account the full 2-dimensional nature of the astrophysical scene and the spatial variation of the Point Spread Function (PSF) due to the optical diffraction from the starshade. The simulations adopt the nominal performance parameters from current S5 results, [17][18][19][20] including the new optical edge coating that reduces the solar glint by a factor of 10. 24 Astrophysical and observational scenarios are selected to represent key science objectives of the well-studied starshade mission concepts including Roman Rendezvous 11 and HabEx 12 ( Table 1). To explore these astrophysical and observational scenarios, as well as key instrument performance parameters (Table 2), approximately 400 images will be simulated.
Participating teams of the data challenge will then develop image-processing algorithms to test the ability to retrieve faint exoplanet signals from the synthetic images and quantify the precision of background calibration. The participating teams will attempt to determine from the images the number of the planets and their locations and brightness, as well as to extract the inclination, density, and potential structures in the exozodiacal dust disk. Estimating the uncertainties of these parameters are essential, because the resulting S/N would be compared with the S/N estimated from idealized exposure time calculators. 16 With the simulated images of slit-prism spectroscopy for Roman and the data cubes of integral field spectroscopy for HabEx, the participating teams will also attempt to extract the planets' spectra. Results from the analyses will determine the detection limit of planets vis-à-vis instrument parameters and indicate how well image-processing algorithms can subtract the background to the photon-noise limit. These results will inform S5 of a realistic noise budget of starshade exoplanet observations and requirements on key instrument performance parameters. The algorithms and science insight gained in the study will be disseminated among science communities via publications and code releases.
The first set of synthetic images has been released to the public in January 2021 and the community participation of the data challenge has started 2 . The data challenge is scheduled to be completed by September 2021. Table 1 lists the astrophysical and observational scenarios adopted as the representative cases for the data challenge. These scenarios are chosen to probe the key and limiting science objectives of Starshade Rendezvous with Roman and HabEx. Four stars will be included in the study, including two stars < 4 parsecs away (τ Ceti and Indi A), one star in the 5 -6-parsec distance range (σ Draconis), and one star in the ∼ 8-parsec distance range (β CVn). These stars are in the nominal target lists of both Starshade Rendezvous with Roman and HabEx. 11,12 29,30 or planets with massive water envelopes. 31, 32 We use Exo-REL, 33 a well-documented model for exoplanet clouds and reflected-light spectra, to simulate the input spectra of the planets.

Astrophysical and Observational Scenarios
The star τ Ceti deserves special attention because of its known planets and outer dust disk.
The analyses of radial-velocity data have not reached a consensus, but the two outer planets (the planets e and f) near its habitable zone are consistent between analyses. 34,35 A debris disk has been detected with far-infrared and radio observations, [36][37][38] with an inner edge at ∼ 6 AU , an outer edge at ∼ 50 AU, and an inclination of ∼ 35 • . The knowledge motivates us to consider the following in the data challenge for τ Ceti. (1) We adopt the debris disk's inclination as the inclination of the planets and the exozodiacal disk. As such, the true masses of the planets e and f are ∼ 6.9 M ⊕ . We further consider the possibility that the planets are either predominantly rocky or with large water envelopes 31 in estimating their radii and simulating the spectra. (2) We include the possibility of a denser exozodiacal disk to test its impact on planet detection ( Table 1). The LBTI exozodiacal disk survey did not detect a disk at τ Ceti, but the 1 − σ upper limit is 44 zodis. 39 (3) We include another planet, with Earth's radius and Earth's mass, between the orbits of the planets e and f. This hypothetical planet is predicted by orbital dynamics and exoplanet population-level information. 40 We verify that the planet would be dynamically stable 41 and would induce a radial-velocity signal amplitude of ∼ 6 cm s −1 , which is well below the detection limit of existing data.
In this data challenge, we will test the ability to detect planets embedded in their exozodiacal disk. Because it is not practical to simulate a self-consistent disk with the planets assumed -given the uncertainties in the source of the particles, the existence of outer planets, as well as the particle size distributions -we instead attempt to bound the problem by considering the endmembers of a "smooth" disk and a "clumpy" disk. For the "smooth" disk we adopt a solar-system disk density profile and use Zodipic 42 to simulate the intensity including the effects of inclination and particle forward scattering. 22 The "clumpy" disk represents a more challenging scenario for planet detec-tion, where clumps of dust particles are trapped in mean-motion resonance with the planets. 21,22 To our knowledge, this would be the first time that potential structures of the exozodiacal disk are assessed against the imaging and detection of small exoplanets of nearby stars. We envision the data challenge would eventually inform us to what extent a clumpy disk would hinder the revelation of the planets in the system.
Lastly, we design the observational scenarios to mimic the basic ideas of operation outlined by Starshade Rendezvous with Roman 11 and HabEx. 12 Starshade Rendezvous with Roman would perform broadband searches of small planets, and if any feasible planets are detected, conduct spectroscopy immediately following imaging in the green band (615-800 nm). 26 We adopt this philosophy by considering two visits per astrophysical scenario, each with both broadband imaging and slit-prism spectroscopy. We will additionally explore two aspects for Starshade Rendezvous with Roman. (1) We will include broadband imaging in the blue band (425-552 nm) and compare it with the green band. The blue band would have less exozodiacal light per resolution element and may thus lead to better planet detection. (2) We will include the green-band spectroscopy up to 850 nm, where the expected instrument contrast degrades from 10 −10 at 800 nm to 10 −9 at 850 nm. 11 This is motivated by the desire to extend coverage to longer wavelengths to reduce spectral degeneracies 43 and the realization that an instrument contrast of 10 −9 may be sufficient. 16 HabEx would conduct most planet searches with its coronagraph and perform integral field spectroscopy with its starshade. 12 We thus focus on spectroscopy for HabEx, also considering two visits per astrophysical scenario. As all scenarios adopted contain planets that fit the definition of "highinterest" by HabEx, we will include near-infrared spectroscopy (1.0-1.8 µm) in addition to the 0.3-1.0 µm band for all scenarios. Note that this entails two spectral integrations per visit, as the starshade needs to be located at a different separation from the telescope for conducting the near-infrared spectroscopy. Because this data challenge focuses on image processing and background calibration, it does not address potential synergy between a starshade and the CGI of Roman, or other operational and logistical constraints of the starshade.   Table 2 lists the instrumental and other effects to be included in the synthetic images and explored by the data challenge. We will adopt the current best estimates for the residual starlight, solar glint, other stray light, and formation flying performance as summarized in Hu et al. (2021). 16 As the residual starlight contrast is the fundamental requirement that controls mechanical precision tolerance, we will vary the contrast level and see how much we could tolerate without adversely impacting planet detection and background calibration. We will adopt the latest estimate of the brightness of the solar glint 24 and also investigate how much brighter we could tolerate. Other stray light sources such as the reflected light of the Milky Way, Earth, Jupiter, as well as the leakage through micrometeoroid holes, will be dimmer than the solar glint by a factor of at least 2 16 and their impact will be studied together with the enhancement factor applied to the solar glint (Table 2).

Instrument Effects
A new effect that will be simulated for the data challenge is the time-dependent variability of these backgrounds. As the solar glint is a starshade's closest analog to speckles in coronagraphic imaging, it is important to include the temporal variability of the stray light to achieve high-fidelity image simulations. We consider telescope jitter, lateral motion of the starshade in formation flight, as well as the change in Sun's angle during long integration as the main sources of temporal variability. For instance, images will be produced with realistic random pointing errors. This way, the data challenge will be able to assess the precision of background calibration achievable from real observations and how it compares with the photon-noise limit.
In addition to broadband imaging, we will simulate spectroscopy with Roman's slit-prism spectrometer and HabEx's integral field spectrometer. The simulations will mimic the dispersion of the planets and background sources and the recording of the spectra on the detector. The spectral resolution, as well as the width and orientation of the slit when applicable, will follow the specifications of the mission concepts. For Roman, we will study whether a specific slit orientation would be necessary or preferred for extracting planetary spectra from interfering backgrounds such as exozodiacal light. For instance, the slit may be oriented radially along the axis between the star and the planet, perpendicular to that axis, or at some angle to both -this may affect the ability to subtract off the exozodiacal background. To our knowledge, it will be the first time that slit-prism spectroscopy will be simulated and studied in detail for exoplanet direct imaging using a starshade.
The knowledge gained in this analysis will help us better understand the background calibration and planet signal extraction in this form of spectroscopy.
Besides the parameters listed in Table 2, we will adopt the telescope and instrument parameters designed for Roman and HabEx, including the optical throughput and detector properties. The frame rate will be chosen for each observation consistent with the photon counting mode of the EMCCDs and the detector noise as the combination of dark current and clock-induced charge will be included accordingly. 15

Summary and Expected Outcomes
To summarize, the Starshade Exoplanet Data Challenge will assess the noise budget of exoplanet observations using a starshade and determine the precision of background calibration achievable with synthetic images. As described in Section 2, the data challenge will be based on the hitherto most realistic simulations of starshade-assisted observations that explore the expected diversity of planet types as well as the density, structure, and inclination of the dust disks around the nearby stars. In addition to an ensemble of instrumental effects, we will include temporal variability of residual starlight and stray light due to the starshade's motion and telescope's jitter. The highfidelity synthetic images will help us better understand the detection limit of planets and their spectra as a function of instrument performance parameters.
Specific outcomes that may be anticipated from the data challenge include (1) estimation of the S/N of planetary parameters (e.g., location, brightness, spectrum) based on the synthetic images.
The estimated S/N, in comparison with the idealized S/N used to set the integration time (Table 2), will tell us how precisely the exozodiacal light and solar glint can be calibrated for planet detection.
(2) Detection of exozodiacal disks and constraints of their density, inclination, and possible clumpy structures. We will assess what we could learn about exozodiacal dust disks from direct-imaging observations using a starshade, and also evaluate whether the clumpy structures would interfere with planet detection. (3) Extraction of planetary spectra, especially from slit-prism spectroscopy.
Using synthetic images, we will evaluate whether it would be feasible to extract planetary spectra when both the planet and the extended background of exozodiacal light are dispersed onto the detector. As a whole, these insights will tell us what we could realistically expect to learn about planets and disks around nearby stars using a starshade, and how these capabilities would depend on the instrument contrast and the suppression of solar glint and other straylight.
The image-processing algorithms and science insight gained in this exercise may also advance high-contrast imaging astronomy in general. Compared to coronagraph direct imaging from the ground and in space, where creative algorithms have been developed to subtract backgrounds and speckles, starshade-assisted direct imaging mainly presents a different set of problems, associated with distinguishing planets from the exozodiacal disk, as well as removing stray light terms specific to the starshade. The data challenge will enable a cross-disciplinary development of techniques to be used to maximize the science yield of the future missions using starshades, ranging from step-by-step (e.g., image subtractions and feature extraction) to more holistic (e.g., Bayesian inversion, deep learning) image analyzing techniques. With the synthetic images, algorithms, science and technology insight, and community partnership, the Starshade Exoplanet Data Challenge will result in an enduring legacy that advances exoplanet astronomy in the years to come.