ZeroBubbleH2Schedule<T>
Models & Types · AiDotNet.DistributedTraining
Implements the Zero Bubble H2 (ZB-H2) pipeline schedule.
Implements: IPipelineSchedule<T>
For Beginners
ZB-H2 is the “maximum throughput” variant. It allows more micro-batches to be in progress simultaneously (using more memory) to completely eliminate idle time. If you have enough GPU memory, ZB-H2 gives the best possible pipeline utilization.
The tradeoff:
- ZB-H1: Same memory as 1F1B, ~1/3 bubble
- ZB-H2: More memory than 1F1B, ~0% bubble (zero idle time)
How It Works
ZB-H2 achieves true zero pipeline bubble by allowing more in-flight micro-batches than 1F1B, trading peak memory for throughput. Like ZB-H1, it splits backward into BackwardInput (B) and BackwardWeight (W), but schedules more aggressively.
Reference: Qi et al., “Zero Bubble Pipeline Parallelism”, ICLR 2024 Spotlight.
https://arxiv.org/abs/2401.10241
Constructors
| Constructor | Summary |
|---|---|
ZeroBubbleH2Schedule() |
Properties
| Property | Summary |
|---|---|
int VirtualStagesPerRank { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches) |
|
T EstimateBubbleFraction(int numStages, int numMicroBatches) |