We review the fault tracking system used by Subaru Telescope operators during night observation, from first light in 1998 to present. Over the years, there was an evolution of format, features and software that culminated in a major revision in 2017. The current revision is LAMP-based using MySQL and PHP, and includes all past faults, pictures and comments. We attempted assigning faults to in-house experts, maintaining a status for each fault from initial report to well-considered solution, multi-language support, displays in both HTML and simple text format, and promotion of more successful solutions over less successful solutions to the same fault. We succeeded in full text search for whole words and substrings, multiple search criteria, and categorization of one fault to two separate subsystems. In the current revision, emphasis was placed on removing obsolete or unused data structures, providing in-memory tabs of the most recently used fault/solutions without repetitive queries, and adding Like/Dislike buttons with cumulative totals for suggesting the most successful fault/solutions across all operators. The current Subaru Fault Tracking System (FATS) is composed of 1440 fault/solutions, 198 pictures, 2453 comments, and is often the first and only resource required in support of night trouble.
Subaru Telescope has recently replaced most equipment of Subaru Telescope Network II with the new equipment which
includes 124TB of RAID system for data archive. Switching the data storage from tape to RAID enables users to access
the data faster. The STN-III dropped some important components of STN-II, such as supercomputers, development &
testing subsystem for Subaru Observation Control System, or data processing subsystem. On the other hand, we invested
more computers to the remote operation system. Thanks to IT innovations, our LAN as well as the network between Hilo
and summit were upgraded to gigabit network at the similar or even reduced cost from the previous system. As the result
of the redesigning of the computer system by more focusing on the observatory operation, we greatly reduced the total
cost for computer rental, purchase and maintenance.
The Subaru Telescope is in process of developing an open-source alternative to the
1st-generation software and databases (STARS 1) used for archiving and query. For
STARS 2, we have chosen PHP and Python for scripting and MySQL as the database
software. We have collected feedback from staff and observers, and used this feedback
to significantly improve the design and functionality of our future archiving and query
Archiving - We identified two weaknesses in 1st-generation STARS archiving software:
a complex and inflexible table structure and uncoordinated system administration for our
business model: taking pictures from the summit and archiving them in both Hawaii and
Japan. We adopted a simplified and normalized table structure with passive keyword
collection, and we are designing an archive-to-archive file transfer system that
automatically reports real-time status and error conditions and permits error recovery.
Query - We identified several weaknesses in 1st-generation STARS query software:
inflexible query tools, poor sharing of calibration data, and no automatic file transfer
mechanisms to observers. We are developing improved query tools and sharing of
calibration data, and multi-protocol unassisted file transfer mechanisms for observers. In
the process, we have redefined a 'query': from an invisible search result that can only
transfer once in-house right now, with little status and error reporting and no error
recovery - to a stored search result that can be monitored, transferred to different
locations with multiple protocols, reporting status and error conditions and permitting
recovery from errors.
More than three years have passed since Subaru Telescope started its Open Use operation. Currently, more than 60% of the total telescope time is spent for scientific observation. Firstly, we define an index to measure how the telescope is effectively used. By using the index, we review the use of the telescope since 2000. Remote observation and queue observation is a long-term goal of Subaru operation because they are believed to be more efficient way to use the telescope and available resource. Control and observation software has been designed and developed to achieve remote observation and queue observation. Currently, about 30% of the telescope time is used as remote observation. We will discuss how much remote observation has contributed to make the use of the telescope effective.
The Subaru Telescope requires a fault tracking system to record the problems and questions that staff experience during their work, and the solutions provided by technical experts to these problems and questions. The system records each fault and routes it to a pre-selected 'solution-provider' for each type of fault. The solution provider analyzes the fault and writes a solution that is routed back to the fault reporter and recorded in a 'knowledge-base' for future reference. The specifications of our fault tracking system were unique. (1) Dual language capacity -- Our staff speak both English and Japanese. Our contractors speak Japanese. (2) Heterogeneous computers -- Our computer workstations are a mixture of SPARCstations, Macintosh and Windows computers. (3) Integration with prime contractors -- Mitsubishi and Fujitsu are primary contractors in the construction of the telescope. In many cases, our 'experts' are our contractors. (4) Operator scheduling -- Our operators spend 50% of their work-month operating the telescope, the other 50% is spent working day shift at the base facility in Hilo, or day shift at the summit. We plan for 8 operators, with a frequent rotation. We need to keep all operators informed on the current status of all faults, no matter the operator's location.