IPipelineSchedule<T>

Interfaces · AiDotNet.Interfaces

Defines a scheduling strategy for pipeline parallel training.

Known implementations: GPipeSchedule<T>, Interleaved1F1BSchedule<T>, LoopedBFSSchedule<T>, OneForwardOneBackwardSchedule<T>, ZeroBubbleH1Schedule<T>, ZeroBubbleH2Schedule<T>, ZeroBubbleVSchedule<T>

For Beginners

In pipeline parallelism, multiple stages process data like an assembly line. A “schedule” decides the order of operations to keep all stages as busy as possible and minimize idle time (“pipeline bubbles”).

Think of it like coordinating workers on an assembly line:

  • GPipe: Worker 1 finishes ALL items, then Worker 2 starts ALL items (simple but slow)
  • 1F1B: Workers alternate between forward and backward steps (more complex but faster)
  • Zero Bubble: Workers split backward into two parts, using the flexible part to fill gaps

How It Works

Pipeline schedules determine the order in which forward and backward passes execute across micro-batches and stages. Different schedules trade off memory usage, pipeline bubble overhead, and implementation complexity.

Schedules fall into two categories:

  • Single-stage: Each rank owns one contiguous model chunk (GPipe, 1F1B, ZB-H1, ZB-H2).
  • Multi-stage: Each rank owns V non-contiguous chunks (“virtual stages”)

(Interleaved 1F1B, Looped BFS, ZB-V).

Properties

Property Summary
int VirtualStagesPerRank { get; } Gets the number of virtual stages (model chunks) each rank holds.
string Name { get; } Gets the name of the scheduling strategy for diagnostics.

Methods

Method Summary
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches) Generates the sequence of operations for a given stage in the pipeline.
T EstimateBubbleFraction(int numStages, int numMicroBatches) Estimates the pipeline bubble fraction for this schedule.