Application of iLauncher interfaces to the abstraction of deep learning workflows across a diversity of computing resources

Rhonda J. Vickery; John Nehrbass; Christopher Moore; Jack Harris; Ryan Larson

doi:10.1117/12.2617437

31 May 2022 Application of iLauncher interfaces to the abstraction of deep learning workflows across a diversity of computing resources

Rhonda J. Vickery, John Nehrbass, Christopher Moore, Jack Harris, Ryan Larson

Author Affiliations +

Proceedings Volume 12095, Algorithms for Synthetic Aperture Radar Imagery XXIX; 1209507 (2022) https://doi.org/10.1117/12.2617437
Event: SPIE Defense + Commercial Sensing, 2022, Orlando, Florida, United States

Abstract

We present some of the latest advancements in the development of the Interface Launcher (iLauncher), along with the application of this technology to the development of plugins that support distributed PyTorch deep learning workflows across a diversity of computing resources including Amazon Web Services (AWS) GovCloud, distributed clusters of heterogeneous nodes with multiple graphics processing units (GPUs) per node running the Slurm batch queuing software, and Department of Defense (DoD) high performance computing (HPC) supercomputers running the Portable Batch Scheduling (PBS) software. The iLauncher technology automates the submission of HPC jobs and provides a mechanism for rapidly prototyping web interfaces from the user’s desktop to powerful capabilities running on the HPC nodes. We describe the extension of previous work to show the development of the client-side plugin JavaScript Object Notation (JSON) description, the underlying server-side scripts for running distributed PyTorch deep learning models on various platforms with different queuing systems, and the recipes for the software along with all dependencies in an all-inclusive software packaging technology called a container. Finally, we show a representative use case running distributed PyTorch in a Jupyter Notebook through iLauncher on the various backend platforms along with some guidance on when each one may be beneficial for a range of scenarios based on models and data.

Conference Presentation

Citation Download Citation

Rhonda J. Vickery, John Nehrbass, Christopher Moore, Jack Harris, and Ryan Larson "Application of iLauncher interfaces to the abstraction of deep learning workflows across a diversity of computing resources", Proc. SPIE 12095, Algorithms for Synthetic Aperture Radar Imagery XXIX, 1209507 (31 May 2022); https://doi.org/10.1117/12.2617437

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available