BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20211207T054806Z
LOCATION:230-231-232
DTSTART;TZID=America/Chicago:20211117T133000
DTEND;TZID=America/Chicago:20211117T140000
UID:submissions.supercomputing.org_SC21_sess174_pap482@linklings.com
SUMMARY:Efficient Large-Scale Language Model Training on GPU Clusters Usin
 g Megatron-LM
DESCRIPTION:Paper\n\nEfficient Large-Scale Language Model Training on GPU 
 Clusters Using Megatron-LM\n\nNarayanan, Shoeybi, Casper, LeGresley, Patwa
 ry...\n\nLarge language models have led to state-of-the-art accuracies acr
 oss several tasks. However, training these models efficiently is challengi
 ng because: a) GPU memory capacity is limited, making it impossible to fit
  large models on even a multi-GPU server, and b) the number of compute ope
 rations required can result in unrealistically long training times. Conseq
 uently, new methods of model parallelism such as tensor and pipeline paral
 lelism have been proposed. Unfortunately, naive usage of these methods lea
 ds to scaling issues at thousands of GPUs. In this paper, we show how tens
 or, pipeline, and data parallelism can be composed to scale to thousands o
 f GPUs. We propose a novel interleaved pipelining schedule that can improv
 e throughput by 10+% with memory footprint comparable to existing approach
 es. Our approach allows us to perform training iterations on a model with 
 1 trillion parameters at 502 petaFLOP/s on 3072 GPUs (per-GPU throughput o
 f 52% of theoretical peak).\n\nTag: Reproducibility Badge, Machine Learnin
 g and Artificial Intelligence\n\nRegistration Category: Tech Program Reg P
 ass\n\nAward Finalist: Best Student Paper Finalists\n\nReproducibility Bad
 ges: Artifact Available, Artifact Functional
END:VEVENT
END:VCALENDAR