Data from two IR survey cameras UKIRT's WFCAM and ESO's VISTA can arrive at rates approaching 1.4 TB/night
for of order 10 years. Handling the rate, and volume of survey data accumulated over time, are both challenges. The
UK's VISTA Data Flow System (for WFCAM & VISTA near-IR survey data) removes instrumental artefacts,
astrometrically and photometrically calibrates, extracts catalogues, puts the products in a curated archive, facilitates
production of user-specified data products, and is designed in the context of the Virtual Observatory. The VDFS design
concept is outlined, and experience in handling the first year of WFCAM data described. This work will minimize risk
in meeting the more taxing requirements of VISTA, which will be commissioned in 2007. Tools for preparing survey
observations with VISTA are outlined.
VISTA Data Flow System (VDFS) survey data products are expected to reach of order one petabyte in volume. Fast and flexible user access to these data is pivotal for efficient science exploitation. In this paper, we describe the provision for survey products archive access and curation which is the final link in the data flow system from telescope to user. Science archive development at the Wide Field Astronomy Unit of the Institute for Astronomy within the University of Edinburgh is taking a phased approach. The first phase VDFS science archive is being implemented for WFCAM, a wide-field infrared imager that has similar output to, but at a lower data rate than the VISTA camera. We describe the WFCAM Science Archive, emphasising the design approach that is intended to lead to a scalable archive system that can handle the huge volume of VISTA data.
Data from the two IR survey cameras WFCAM (at UKIRT in the northern hemisphere) and VISTA (at ESO in the southern hemisphere) can arrive at rates approaching 1.4 TB/night for of order 10 years. Handling the data rates on a nightly basis, and the volumes of survey data accumulated over time each present new challenges. The approach adopted by the UK's VISTA Data Flow System (for WFCAM & VISTA data) is outlined, emphasizing how the design will meet the end-to-end requirements of the system, from on-site monitoring of the quality of the data acquired, removal of instrumental artefacts, astrometric and photometric calibration, to accessibility of curated and user-specified data products in the context of the Virtual Observatory. Accompanying papers by Irwin et al and Hambly et al detail the design of the pipeline and science archive aspects of the project.
The UKIRT Wide Field Camera (WFCAM) is an IR mosaic camera that represents an enormous leap in deep IR survey capability. It will be used as both an open time facility, and to perform a public IR Deep Sky Survey (the UKIDSS project), starting in early 2004. Here we present current plans for the data archive system, which will be provided as a standard service for all UK WFCAM data whether private or public survey data. The data rate is an order of magnitude larger than any previous survey experiment. WFCAM is therefore a crucial stepping stone between current day surveys such as SuperCOSMOS, APM and SDSS, and future facilities such as VISTA and the LSST. Pipeline processing presents a technical challenge, but the strongest challenges come in operation and curation of such a pipeline and of the rapidly accumulating database. For the public archive, there is little technical challenge in simply storing the data, and the real challenge comes in the rapidly increasing expectations of the user community for the kind of on-line services available with the archive. We describe three levels of archive service and the challenges they present, and discuss the hardware and software solutions we are likely to deploy.