Application of Jupyter Notebook interfaces and iLauncher to deep learning workflows on HPC systems

Rhonda J. Vickery; John Nehrbass; Daniel Mogilevsky; Jack Harris; Ryan Larson

doi:10.1117/12.2586092

12 April 2021 Application of Jupyter Notebook interfaces and iLauncher to deep learning workflows on HPC systems

Rhonda J. Vickery, John Nehrbass, Daniel Mogilevsky, Jack Harris, Ryan Larson

Proceedings Volume 11728, Algorithms for Synthetic Aperture Radar Imagery XXVIII; 117280D (2021) https://doi.org/10.1117/12.2586092
Event: SPIE Defense + Commercial Sensing, 2021, Online Only

Abstract

We present information on what users need in order to support complex deep learning workflows on DoD HPC systems with web interfaces using ipywidgets in Jupyter Notebooks and the Interface Launcher (iLauncher), a tool that automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We detail a representative use case for a PyTorch deep learning workflow and show how to include the underlying software along with all dependencies in an all-inclusive software packaging technology called a Singularity container. We then show how to use ipywidgets in a Jupyter Notebook and convert it to a full-fledged web interface using the Voila server. Finally, we outline how to create the iLauncher plugin to run the web interface on DoD HPC system nodes to provide a complete user interface workflow solution that does not require special privileges to create, deploy, or use. Using Jupyter Notebooks, ipywidgets, iLauncher, and Singularity containers together provide an explosion of accessible capabilities that were previously inconceivable in the restrictive DoD environment.

Conference Presentation

Citation Download Citation

Rhonda J. Vickery, John Nehrbass, Daniel Mogilevsky, Jack Harris, and Ryan Larson "Application of Jupyter Notebook interfaces and iLauncher to deep learning workflows on HPC systems", Proc. SPIE 11728, Algorithms for Synthetic Aperture Radar Imagery XXVIII, 117280D (12 April 2021); https://doi.org/10.1117/12.2586092

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available