Talk
Large-scale model training rarely fails the way engineers expect it to. The intuition borrowed from other domains treats it as a batch job - provision the hardware, launch the run, wait for convergence - but published accounts from Meta's Llama 3 pre-training and ByteDance's training infrastructure describe interruptions every few hours and thousands of jobs per quarter that continue holding GPUs and incurring cost long after progress has stopped. The costly failures are usually not crashes, but runs that look healthy while computing nothing.
Part of the reason is that the most familiar metric misleads. GPU utilization indicates only that a kernel was resident during the sample window, so a stalled job can report near-total engine activity alongside almost no work in the streaming multiprocessors, and every dashboard stays green.
This talk goes through five failure profiles observed on real training clusters - checkpoint stalls, distributed hangs, stragglers, communication-bound runs, and silently throttled GPUs - showing for each what the telemetry actually looks like and which standard tooling exposes it, from DCGM counters and NCCL debug output to the PyTorch flight recorder, eBPF probes, and host stack sampling.