18 July 2014 An experiment in big data: storage, querying and visualisation of data taken from the Liverpool Telescope's wide field cameras
Author Affiliations +
Abstract
The Small Telescopes Installed at the Liverpool Telescope (STILT) project has been in operation since March 2009, collecting data with three wide field unfiltered cameras: SkycamA, SkycamT and SkycamZ. To process the data, a pipeline was developed to automate source extraction, catalogue cross-matching, photometric calibration and database storage. In this paper, modifications and further developments to this pipeline will be discussed, including a complete refactor of the pipeline's codebase into Python, migration of the back-end database technology from MySQL to PostgreSQL, and changing the catalogue used for source cross-matching from USNO-B1 to APASS. In addition to this, details will be given relating to the development of a preliminary front-end to the source extracted database which will allow a user to perform common queries such as cone searches and light curve comparisons of catalogue and non-catalogue matched objects. Some next steps and future ideas for the project will also be presented.
© (2014) COPYRIGHT Society of Photo-Optical Instrumentation Engineers (SPIE). Downloading of the abstract is permitted for personal use only.
R. M. Barnsley, R. M. Barnsley, Iain A. Steele, Iain A. Steele, R. J. Smith, R. J. Smith, Neil R. Mawson, Neil R. Mawson, } "An experiment in big data: storage, querying and visualisation of data taken from the Liverpool Telescope's wide field cameras", Proc. SPIE 9152, Software and Cyberinfrastructure for Astronomy III, 91522L (18 July 2014); doi: 10.1117/12.2055558; https://doi.org/10.1117/12.2055558
PROCEEDINGS
10 PAGES


SHARE
RELATED CONTENT

Data processing factory for the Sloan Digital Sky Survey
Proceedings of SPIE (December 23 2002)
J-PAS data management pipeline and archiving
Proceedings of SPIE (September 23 2012)
Building archives in the virtual observatory era
Proceedings of SPIE (July 19 2010)
Korea microlensing telescope network: data management plan
Proceedings of SPIE (September 23 2012)

Back to Top