ZeroBubbleH2Schedule<T>

Models & Types · AiDotNet.DistributedTraining

Implements the Zero Bubble H2 (ZB-H2) pipeline schedule.

Implements: IPipelineSchedule<T>

For Beginners

ZB-H2 is the “maximum throughput” variant. It allows more micro-batches to be in progress simultaneously (using more memory) to completely eliminate idle time. If you have enough GPU memory, ZB-H2 gives the best possible pipeline utilization.

The tradeoff:

  • ZB-H1: Same memory as 1F1B, ~1/3 bubble
  • ZB-H2: More memory than 1F1B, ~0% bubble (zero idle time)

How It Works

ZB-H2 achieves true zero pipeline bubble by allowing more in-flight micro-batches than 1F1B, trading peak memory for throughput. Like ZB-H1, it splits backward into BackwardInput (B) and BackwardWeight (W), but schedules more aggressively.

Reference: Qi et al., “Zero Bubble Pipeline Parallelism”, ICLR 2024 Spotlight. https://arxiv.org/abs/2401.10241

Constructors

Constructor Summary
ZeroBubbleH2Schedule()

Properties

Property Summary
int VirtualStagesPerRank { get; }
string Name { get; }

Methods

Method Summary
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches)
T EstimateBubbleFraction(int numStages, int numMicroBatches)