PipelineParallelModel<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements Pipeline Parallel model wrapper - splits model into stages across ranks.
Inheritance: ShardedModelBase<T, TInput, TOutput> → PipelineParallelModel<T, TInput, TOutput>
For Beginners
Pipeline parallelism is like an assembly line for training. Imagine a deep neural network as a tall building - instead of one person (GPU) handling all floors, we assign different floors to different people. Process 0 handles layers 0-10, Process 1 handles layers 11-20, etc.
To keep everyone busy (avoid idle time), we split each batch into smaller “micro-batches” that flow through the pipeline like cars on an assembly line. While Process 1 is working on micro-batch 1, Process 0 can start on micro-batch 2.
How It Works
Strategy Overview: Pipeline Parallelism divides the model vertically into stages, with each process owning specific layers. Input mini-batches are divided into micro-batches that flow through the pipeline stages sequentially. This enables training models too large to fit on a single device while maintaining good hardware utilization through micro-batch pipelining.
Supported Features (Issue #463):
- 7 Pipeline Schedules: GPipe, 1F1B, ZB-H1, ZB-H2, ZB-V, Interleaved 1F1B, Looped BFS.
Zero Bubble schedules decompose backward into BackwardInput + BackwardWeight for optimal throughput.
- Virtual Stages: Multi-stage schedules (Interleaved 1F1B, Looped BFS, ZB-V) assign
multiple non-contiguous model chunks per rank, reducing pipeline bubble by factor V.
- Micro-Batch Slicing: Input is automatically sliced into micro-batches that flow
through the pipeline independently.
- Backward Decomposition: If the wrapped model implements
IPipelineDecomposableModel,
BackwardInput and BackwardWeight are truly decomposed. Otherwise, a compatible emulation is used.
- Activation Checkpointing: Trade compute for memory by recomputing activations from
checkpoints during the backward pass.
- Load-Balanced Partitioning: Balance compute across stages via dynamic programming.
Example
// Wrap an existing model for pipeline-parallel training across multiple ranks
var architecture = new NeuralNetworkArchitecture<float>(
InputType.OneDimensional, NeuralNetworkTaskType.Regression,
inputSize: 8, outputSize: 1);
var wrappedModel = new NeuralNetwork<float>(architecture);
var backend = new InMemoryCommunicationBackend<float>(rank: 0, worldSize: 2);
var config = new ShardingConfiguration<float>(backend);
var schedule = new GPipeSchedule<float>();
var pipelineModel = new PipelineParallelModel<float, Tensor<float>, Tensor<float>>(
wrappedModel, config, microBatchCount: 4, schedule: schedule);
Constructors
| Constructor | Summary |
|---|---|
PipelineParallelModel(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config, int microBatchCount, IPipelinePartitionStrategy<T> partitionStrategy, IPipelineSchedule<T> schedule, ActivationCheckpointConfig checkpointConfig) |
Creates a new Pipeline Parallel model. |
Properties
| Property | Summary |
|---|---|
T EstimatedBubbleFraction { get; } |
Gets the estimated pipeline bubble fraction for the current configuration. |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters) |
|
ModelMetadata<T> GetModelMetadata() |
|
TOutput Predict(TInput input) |
|
void LoadModel(string filePath) |
|
void SaveModel(string filePath) |
|
void Train(TInput input, TOutput expectedOutput) |