BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20211207T054800Z
LOCATION:230-231-232
DTSTART;TZID=America/Chicago:20211116T143000
DTEND;TZID=America/Chicago:20211116T150000
UID:submissions.supercomputing.org_SC21_sess172_pap143@linklings.com
SUMMARY:Chimera: Efficiently Training Large-Scale Neural Networks with Bid
 irectional Pipelines
DESCRIPTION:Paper\n\nChimera: Efficiently Training Large-Scale Neural Netw
 orks with Bidirectional Pipelines\n\nLi, Hoefler\n\nTraining large deep le
 arning models at scale is very challenging. This paper proposes Chimera, a
  novel pipeline parallelism scheme which combines bidirectional pipelines 
 for efficiently training large-scale models. Chimera is a synchronous appr
 oach and therefore no loss of accuracy, which is more convergence-friendly
  than asynchronous approaches. Compared with the latest synchronous pipeli
 ne approach, Chimera reduces the number of bubbles by up to 50%; benefitin
 g from the sophisticated scheduling of bidirectional pipelines, Chimera ha
 s a more balanced activation memory consumption. Evaluations are conducted
  on Transformer based language models. For a GPT-2 model with 1.3 billion 
 parameters running on 2,048 GPU nodes of the Piz Daint supercomputer, Chim
 era improves the training throughput by 1.16x-2.34x over the state-of-the-
 art synchronous and asynchronous pipeline approaches.\n\nTag: Reproducibil
 ity Badge, Machine Learning and Artificial Intelligence\n\nRegistration Ca
 tegory: Tech Program Reg Pass\n\nAward Finalist: Best Paper Finalist\n\nRe
 producibility Badges: Artifact Functional
END:VEVENT
END:VCALENDAR