BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20211207T055342Z
LOCATION:227
DTSTART;TZID=America/Chicago:20211115T140000
DTEND;TZID=America/Chicago:20211115T143000
UID:submissions.supercomputing.org_SC21_sess347_ws_scsc103@linklings.com
SUMMARY:Evaluating Multi-Level Checkpointing for Distributed Deep Neural N
 etwork Training
DESCRIPTION:Workshop\n\nEvaluating Multi-Level Checkpointing for Distribut
 ed Deep Neural Network Training\n\nAnthony, Dai\n\nDeep learning (DL) appl
 ications are becoming one of the most important applications for HPC and c
 loud systems. The massive datasets and deep neural networks (DNN) used by 
 DL applications introduce many HPC challenges. Therefore, HPC checkpoint/r
 estart tools are an attractive choice. However, most data-parallel DL trai
 ning jobs use a naive scheme called root checkpointing, which is subject t
 o blocking semantics and straggling forward progress. In this work, we app
 ly a multi-level checkpointing tool (SCR-Exa) to distributed DL applicatio
 ns. We examine the performance of two DNN models at scale on Lassen (a lea
 ding TOP500 system), while ensuring the DNN's accuracy is maintained after
  restart from simulated system failures. Our test results show that multi-
 level checkpointing schemes are able to achieve nearly constant overhead a
 t scale. To the best of our knowledge, this study presents the first evalu
 ation to demonstrate strong scalability of a checkpointing scheme for dist
 ributed DL without making framework-specific changes.\n\nTag: Parallel Pro
 gramming Languages and Models, Reliability and Resiliency\n\nRegistration 
 Category: Workshop Reg Pass
END:VEVENT
END:VCALENDAR