A container packages up an application's code and dependencies, so it runs quickly and reliably across all computing environments. Using Singularity, a text-based recipe file is written in order to construct a container. However, all shell commands to install the desired applications and dependencies must be known, creating a daunting task for a novice builder. We solve this using a Python script with a database of installation commands for a plethora of applications, written automatically into a Singularity recipe file. Future work includes expanding the library of available apps, automating the database upkeep, and integrating the script with HPC systems.
This work builds on another effort described in Application of Jupyter Notebook interfaces and iLauncher to deep learning work ows on HPC systems.22 We describe a complex work ow application which generates millions of images in parallel on an HPC system via web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher). Some computations are so complicated, taking many millions of HPC hours, that only a few subject matter experts are able to generate information efficiently. We present our custom application that walks the user through a work flow to include: target selection, configuration of the target, radar phase history simulation, and finally SAR image generation. The interface requests the user to enter a minimal set of parameters while other variables essential to computations are generated on the y and provides status updates on work ow computations. Additionally, the ability to download any data component or view images interactively is provided. This application can be disconnected from the HPC system and reconnected at any time without slowing down the computations on the work ow submitted. Although typically a maximum run time must be specified when submitting a job to the queuing interface on an HPC system, this application uses the HPC-GPS tool to allow users to extend run times even after the initial request is submitted. Our new application helps to reduce the barrier to entry for both complicated physics-based simulations and using HPC systems.
We present information on what users need in order to support complex deep learning workflows on DoD HPC systems with web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher), a tool that automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We detail a representative use case for a PyTorch deep learning workflow and show how to include the underlying software along with all dependencies in an all-inclusive software packaging technology called a Singularity container. We then show how to use ipywidgets in a Jupyter Notebook and convert it to a full-fledged web interface using the Voila server. Finally, we outline how to create the iLauncher plugin to run the web interface on DoD HPC system nodes to provide a complete user interface workflow solution that does not require special privileges to create, deploy, or use. Using Jupyter Notebooks, ipywidgets, iLauncher, and Singularity containers together provide an explosion of accessible capabilities that were previously inconceivable in the restrictive DoD environment.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.