OneForwardOneBackwardSchedule<T>
Models & Types · AiDotNet.DistributedTraining
Implements the 1F1B (One-Forward-One-Backward) pipeline schedule.
Implements: IPipelineSchedule<T>
For Beginners
Instead of doing ALL forward passes then ALL backward passes (GPipe), 1F1B interleaves them. This is like a factory where each worker handles their current item and immediately starts the return processing, rather than waiting for all items to pass through.
Benefits:
- Reduces pipeline bubble from ~50% to ~12-15%
- Limits peak memory to (numStages) stored activations instead of (numMicroBatches)
- More efficient for large numbers of micro-batches
Example with 4 stages and 8 micro-batches:
How It Works
The 1F1B schedule interleaves forward and backward passes to minimize pipeline bubble and memory usage. It has three phases:
- Warmup: Each stage executes forward passes to fill the pipeline.
Stage i performs (numStages - 1 - i) forward passes before steady state.
- Steady State: Each stage alternates between one forward and one backward pass.
This keeps all stages busy and limits memory usage to at most (numStages) activations.
- Cooldown: Remaining backward passes drain the pipeline.
Reference: Narayanan et al., “PipeDream: Generalized Pipeline Parallelism for DNN Training”, SOSP 2019.
https://arxiv.org/abs/1806.03377
Constructors
| Constructor | Summary |
|---|---|
OneForwardOneBackwardSchedule() |
Properties
| Property | Summary |
|---|---|
int VirtualStagesPerRank { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches) |
|
T EstimateBubbleFraction(int numStages, int numMicroBatches) |