The archive of the La Silla Paranal Observatory is a powerful science resource for the ESO astronomical community. It stores both the raw data generated by all ESO instruments and selected processed (science-ready) data. We present the new capabilities and user services that have recently been developed in order to enhance data discovery and usage in the face of the increasing volume and complexity of the archive holdings. Future plans to extend the new services to processed data from the Atacama Large Millimeter/submillimeter Array (ALMA) are also discussed.
Until recently, users of ESO’s Very Large Telescope had to prepare Observing Blocks (OBs) with a standalone desktop tool. Tool support for automated OB mass production was mostly limited to imaging public surveys. Furthermore, there was no connection between the OB preparation software and other ancillary tools, such as Exposure Time Calculators, finding chart preparation software, and observatory schedule, meaning that users had to re-type the same information in several tools, and could design observations that would be incompatible with the Service Mode schedule. To address these shortcomings, we have implemented a new programming interface (API) and a state-of-the-art web application which provide observers with unprecedented flexibility and promote the usage of instrument and science-case specific tools, from small scripts to full-blown user interfaces. In this paper, we describe the software architecture of our solution, important design concepts and the technology stack adopted. We report on first user experience in both Visitor and Service Mode. We discuss tailored API programming examples, solving specific user requirements, and explain API usage scenarios for the next generation of ESO instruments. Finally, we describe the future evolution of our new approach.
Monitoring and prediction of astronomical observing conditions are essential for planning and optimizing observations. For this purpose, ESO, in the 90s, developed the concept of an Astronomical Site Monitor (ASM), as a facility fully integrated in the operations of the VLT observatory<sub>. </sub>Identical systems were installed at Paranal and La Silla, providing comprehensive local weather information. By now, we had very good reasons for a major upgrade: <p> </p>• The need of introducing new features to satisfy the requirements of observing with the Adaptive Optics Facility and to benefit other Adaptive Optics systems. <p> </p>• Managing hardware and software obsolescence. <p> </p>• Making the system more maintainable and expandable by integrating off-the-shelf hardware solutions. <p> </p>The new ASM integrates:<p> </p> • A new Differential Image Motion Monitor (DIMM) paired with a Multi Aperture Scintillation Sensor (MASS) to measure the vertical distribution of turbulence in the high atmosphere and its characteristic velocity. <p> </p>• A new SLOpe Detection And Ranging (SLODAR) telescope, for measuring the altitude and intensity of turbulent layers in the low atmosphere. <p> </p>• A water vapour radiometer to monitor the water vapour content of the atmosphere. <p> </p>• The old weather tower, which is being refurbished with new sensors. The telescopes and the devices integrated are commercial products and we have used as much as possible the control system from the vendors. The existing external interfaces, based on the VLT standards, have been maintained for full backward compatibility. All data produced by the system are directly fed in real time into a relational database. A completely new web-based display replaces the obsolete plots based on HP-UX RTAP. We analyse here the architectural and technological choices and discuss the motivations and trade-offs.
The European Southern Observatory Science Archive Facility is evolving from an archive containing predominantly raw data into a resource also offering science-grade data products for immediate analysis and prompt interpretation. New products originate from two different sources. On the one hand Principal Investigators of Public Surveys and other programmes reduce the raw observational data and return their products using the so-called Phase 3 - a process that extends the Data Flow System after proposal submission (Phase 1) and detailed specification of the observations (Phase 2). On the other hand raw data of selected instruments and modes are uniformly processed in-house, independently of the original science goal. Current data products assets in the ESO science archive facility include calibrated images and spectra, as well as catalogues, for a total volume in excess of 16 TB and increasing. Images alone cover more than 4500 square degrees in the NIR bands and 2400 square degrees in the optical bands; over 85000 individually searchable spectra are already available in the spectroscopic data collection. In this paper we review the evolution of the ESO science archive facility content, illustrate the data access by the community, give an overview of the implemented processes and the role of the associated data standard.
We present the data model utilised in maintaining the lifecycle of astronomical frames in the ESO Archive
activities. The principal concept is that complete file metadata are managed separately from the data and
merged only upon delivery of the data to the end user. This concept is now applied to all ESO Archive assets:
raw observation frames originated in ESO telescopes in all Chilean sites, reduced frames generated intra-ESO
using pipeline processing, as well as the processed data generated by the PIs and delivered to the ESO Archive
through "Phase 3" infrastructure. We present the implementation details of the model and discuss future
Throughout the course of many years of observations at the VLT, the phase 2 software applications supporting the
specification, execution and reporting of observations have been continuously improved and refined. Specifically the
introduction of astronomical surveys propelled the creation of new tools to express more sophisticated, longer-term
observing strategies often consisting of several hundreds of observations. During the execution phase, such survey
programs compete with other service and visitor mode observations and a number of constraints have to be considered.
In order to maximize telescope utilization and execute all programs in a fair way, new algorithms have been developed to
prioritize observable OBs taking into account both current and future constraints (e.g. OB time constraints, technical
telescope time) and suggest the next OB to be executed. As a side effect, a higher degree of observation automation
enables operators to run telescopes mostly autonomously with little supervision by a support astronomer. We describe
the new tools that have been deployed and the iterative and incremental software development process applied to
develop them. We present our key software technologies used so far and discuss potential future evolution both in terms
of features as well as software technologies.
The start of operations of the VISTA survey telescope will not only offer a new facility to the ESO community, but also
a new way of observing. Survey observation programs typically observe large areas of the sky and might span several
years, corresponding to the execution of hundreds of observations blocks (OBs) in service mode. However, the execution
time of an individual survey OB will often be rather short. We expect that up to twelve OBs may be executed per hour,
as opposed to about one OB per hour on ESO's Very Large Telescope (VLT). OBs of different programs are competing
for observation time and must be executed with adequate priority. For these reasons, the scheduling of survey OBs is
required to be almost fully automated. Two new key concepts are introduced to address these challenges: ESO's phase 2
proposal preparation tool P2PP allows PIs of survey programs to express advanced mid-term observing strategies using
scheduling containers of OBs (groups, timelinks, concatenations). Telescope operators are provided with effective short-term
decision support based on ranking observable OBs. The ranking takes into account both empirical probability
distributions of various constraints and the observing strategy described by the scheduling containers. We introduce the
three scheduling container types and describe how survey OBs are ranked. We demonstrate how the new concepts are
implemented in the preparation and observing tools and give an overview of the end-to-end workflow.
The European Organisation for Astronomical Research in the Southern Hemisphere (ESO), headquartered in Garching,
Germany, operates different state-of-the-art observing sites in Chile. To manage observatory operations and observation
transfer, ESO developed an end-to-end Data Flow System, from Phase I proposal preparation to the final archiving of
quality-controlled science, calibration and engineering data. All information pertinent to the data flow is stored in the
central databases at ESO headquarters and replicated to and from the observatory database servers.
In the ESO's data flow model one can distinguish two groups of databases; the front-end databases, which are replicated
from the ESO headquarters to the observing sites, and the back-end databases, where replication is directed from the
observations to the headquarters.
A part of the front-end database contains the Observation Blocks (OBs), which are sequences of operations necessary to
perform an observation, such as instrument setting, target, filter and/or grism ID, exposure time, etc. Observatory
operations rely on fast access to the OB database and quick recovery strategies in case of a database outage.
After several years of operations, those databases have grown considerably. There was a necessity in reviewing the
database architecture to find a solution that support scalability of the operational databases.
We present the newly developed concept of distributing the OBs between two databases, containing operational and
historical information. We present the architectural design in which OBs in operational databases will be archived
periodically at ESO headquarters. This will remedy the scalability problems and keep the size of the operational
databases small. The historical databases will only exist in the headquarters, for archiving purposes.