Interleaved1F1BSchedule<T>

Models & Types · AiDotNet.DistributedTraining

Implements the Interleaved 1F1B pipeline schedule with multiple virtual stages per rank.

Implements: IPipelineSchedule<T>

For Beginners

Standard 1F1B gives each GPU one big chunk of the model. Interleaved 1F1B gives each GPU V smaller, evenly-spaced chunks instead.

Example with 4 GPUs, V=2 (8 total chunks):

  • GPU 0: chunks 0 and 4
  • GPU 1: chunks 1 and 5
  • GPU 2: chunks 2 and 6
  • GPU 3: chunks 3 and 7

This means each microbatch visits each GPU twice (once for each chunk), creating more opportunities to interleave work and reduce idle time. The bubble shrinks from ~(P-1)/(2M+P-1) to ~(P-1)/(2MV+P-1).

Used in production by Megatron-LM v2 and NVIDIA NeMo.

How It Works

Interleaved 1F1B assigns V non-contiguous model chunks (“virtual stages”) to each rank. Rank i holds chunks {i, i+P, i+2P, …} where P is the number of physical ranks. This reduces the pipeline bubble by a factor of V compared to standard 1F1B.

When a microbatch is ready for multiple local virtual stages, Interleaved 1F1B prioritizes the earlier microbatch (depth-first ordering). This is in contrast to Looped BFS which prioritizes the earlier stage.

Reference: Narayanan et al., “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM”, SC 2021. https://arxiv.org/abs/2104.04473

Constructors

Constructor Summary
Interleaved1F1BSchedule(int virtualStagesPerRank) Creates a new Interleaved 1F1B schedule.

Properties

Property Summary
int VirtualStagesPerRank { get; }
string Name { get; }

Methods

Method Summary
IReadOnlyList<PipelineOperation> GetSchedule(int stageId, int numStages, int numMicroBatches)
T EstimateBubbleFraction(int numStages, int numMicroBatches)