ZeroBubbleH1Schedule<T>

Models & Types · AiDotNet.DistributedTraining

Implements the Zero Bubble H1 (ZB-H1) pipeline schedule.

Implements: IPipelineSchedule<T>

For Beginners

In standard 1F1B, the backward pass computes both activation and weight gradients together. ZB-H1 splits this into two steps. The activation gradient (B) must be done quickly (the previous stage is waiting), but the weight gradient (W) can wait. By scheduling W during idle time, we reduce wasted time by ~67% compared to 1F1B.

Think of it like a car wash: the “rinse” (B) must happen right after soap, but “waxing” (W) can be done whenever there’s a free slot.

How It Works

ZB-H1 splits the backward pass into two independent computations:

  • B (BackwardInput): Computes activation gradients (dL/dInput) - on the critical path.
  • W (BackwardWeight): Computes weight gradients (dL/dWeights) - can be deferred.

By deferring W to fill pipeline bubbles, ZB-H1 reduces the bubble to approximately one-third of 1F1B’s bubble while maintaining the same peak memory footprint.

Reference: Qi et al., “Zero Bubble Pipeline Parallelism”, ICLR 2024 Spotlight. https://arxiv.org/abs/2401.10241

Constructors

Constructor Summary
ZeroBubbleH1Schedule()

Properties

Property Summary
int VirtualStagesPerRank { get; }
string Name { get; }

Methods

Method Summary
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches)
T EstimateBubbleFraction(int numStages, int numMicroBatches)