Maksim Afanasyev

Distributed training has a tooling problem. Mainstream distributed training frameworks tend to be monolithic platforms optimized for predefined regimes - powerful, but incredibly difficult to adapt for experimental workflows. Conversely, creating a pipeline from scratch requires deeper expertise, dealing with low-level primitives and writing complex logic from scratch.

This talk introduces d9d, designed to occupy the sweet spot between monoliths and homebrew scripts. We will look under the hood of d9d to uncover the modern engineering paradigms and PyTorch 2.0 features that make robust, hackable scale-out LLM training possible.

Key takeaways

1. Modern Distributed Training 101. A quick introduction to how modern LLMs are trained in a distributed manner, including an overview of 6D parallelism.
2. The "Why?". Tooling for distributed training that is currently available, and why we decided to build a completely new framework.
3. Philosophy of d9d. Key decisions we made during the development process. How we integrate with existing PyTorch 2.0 infrastructure. Why we abandon massive wrappers like DistributedDataParallel. The "white-box" modelling approach. How to seamlessly integrate non-native, high-performance Triton/CUDA extensions and still keep the framework's code readable.
4. Highlighted APIs. Some APIs in d9d we find especially useful and novel: streaming distributed checkpointing engine supporting graph-based model state transformations, refactored pipeline parallelism engine based on torch.distributed.pipelining.
5. Engineering for ML Frameworks. How strict linting, static type-checking, and a comprehensive approach to local and distributed testing keep a highly complex framework reliable.

Attendees will leave with a solid understanding of modern distributed training mechanics, a ready-to-use framework to scale out their LLM research, and actionable software engineering patterns to apply to their own complex Python systems.