ZeroBubbleH1Schedule<T>
Models & Types · AiDotNet.DistributedTraining
Implements the Zero Bubble H1 (ZB-H1) pipeline schedule.
Implements: IPipelineSchedule<T>
For Beginners
In standard 1F1B, the backward pass computes both activation and weight gradients together. ZB-H1 splits this into two steps. The activation gradient (B) must be done quickly (the previous stage is waiting), but the weight gradient (W) can wait. By scheduling W during idle time, we reduce wasted time by ~67% compared to 1F1B.
Think of it like a car wash: the “rinse” (B) must happen right after soap, but “waxing” (W) can be done whenever there’s a free slot.
How It Works
ZB-H1 splits the backward pass into two independent computations:
- B (BackwardInput): Computes activation gradients (dL/dInput) - on the critical path.
- W (BackwardWeight): Computes weight gradients (dL/dWeights) - can be deferred.
By deferring W to fill pipeline bubbles, ZB-H1 reduces the bubble to approximately one-third of 1F1B’s bubble while maintaining the same peak memory footprint.
Reference: Qi et al., “Zero Bubble Pipeline Parallelism”, ICLR 2024 Spotlight.
https://arxiv.org/abs/2401.10241
Constructors
| Constructor | Summary |
|---|---|
ZeroBubbleH1Schedule() |
Properties
| Property | Summary |
|---|---|
int VirtualStagesPerRank { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches) |
|
T EstimateBubbleFraction(int numStages, int numMicroBatches) |