GPipeSchedule<T>
Models & Types · AiDotNet.DistributedTraining
Implements the GPipe scheduling strategy: all forward passes first, then all backward passes.
Implements: IPipelineSchedule<T>
For Beginners
GPipe is the straightforward approach:
- Push ALL micro-batches through the forward pass (left to right through stages)
- Then push ALL micro-batches through the backward pass (right to left)
This creates a “bubble” where stages are idle during pipeline fill and drain. With P stages and M micro-batches, the bubble fraction is approximately (P-1)/(P-1+M).
For 4 stages and 4 micro-batches:
The underscores represent idle time (bubble).
How It Works
GPipe is the simplest pipeline schedule. It executes all forward micro-batches sequentially through the pipeline, storing all activations, then executes all backward micro-batches in reverse order.
Reference: Huang et al., “GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism”, 2019.
https://arxiv.org/abs/1811.06965
Constructors
| Constructor | Summary |
|---|---|
GPipeSchedule() |
Properties
| Property | Summary |
|---|---|
int VirtualStagesPerRank { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches) |
|
T EstimateBubbleFraction(int numStages, int numMicroBatches) |