System optimizations for enabling training of extreme long sequence transformer models

DOI10.1145/3662158.3662806MaRDI QIDQ6971971zbMATH OpenFDO

Authors Sam Ade Jacobs, Masahiro Tanaka, Chengming Zhang, Minjia Zhang, Reza Yazdani Aminadabi, Samyam Rajbhandari, Yuxiong He

Publication date 13 June 2025

Full work available at URL https://doi.org/10.1145/3662158.3662806

zbMATH Keywords

AI/MLdistributed trainingHPClong context LLM

Mathematics Subject Classification ID

Distributed systems (68M14) Distributed algorithms (68W15)

This page was built for publication: System optimizations for enabling training of extreme long sequence transformer models

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6971971)