PipelineParallelModel<T, TInput, TOutput>

Models & Types · AiDotNet.DistributedTraining

Implements Pipeline Parallel model wrapper - splits model into stages across ranks.

Inheritance: ShardedModelBase<T, TInput, TOutput> → PipelineParallelModel<T, TInput, TOutput>

For Beginners

Pipeline parallelism is like an assembly line for training. Imagine a deep neural network as a tall building - instead of one person (GPU) handling all floors, we assign different floors to different people. Process 0 handles layers 0-10, Process 1 handles layers 11-20, etc.

To keep everyone busy (avoid idle time), we split each batch into smaller “micro-batches” that flow through the pipeline like cars on an assembly line. While Process 1 is working on micro-batch 1, Process 0 can start on micro-batch 2.

How It Works

Strategy Overview: Pipeline Parallelism divides the model vertically into stages, with each process owning specific layers. Input mini-batches are divided into micro-batches that flow through the pipeline stages sequentially. This enables training models too large to fit on a single device while maintaining good hardware utilization through micro-batch pipelining.

Supported Features (Issue #463):

  • 7 Pipeline Schedules: GPipe, 1F1B, ZB-H1, ZB-H2, ZB-V, Interleaved 1F1B, Looped BFS.

Zero Bubble schedules decompose backward into BackwardInput + BackwardWeight for optimal throughput.

  • Virtual Stages: Multi-stage schedules (Interleaved 1F1B, Looped BFS, ZB-V) assign

multiple non-contiguous model chunks per rank, reducing pipeline bubble by factor V.

  • Micro-Batch Slicing: Input is automatically sliced into micro-batches that flow

through the pipeline independently.

BackwardInput and BackwardWeight are truly decomposed. Otherwise, a compatible emulation is used.

  • Activation Checkpointing: Trade compute for memory by recomputing activations from

checkpoints during the backward pass.

  • Load-Balanced Partitioning: Balance compute across stages via dynamic programming.

Example

// Wrap an existing model for pipeline-parallel training across multiple ranks
var architecture = new NeuralNetworkArchitecture<float>(
    InputType.OneDimensional, NeuralNetworkTaskType.Regression,
    inputSize: 8, outputSize: 1);
var wrappedModel = new NeuralNetwork<float>(architecture);
            
var backend = new InMemoryCommunicationBackend<float>(rank: 0, worldSize: 2);
var config = new ShardingConfiguration<float>(backend);
var schedule = new GPipeSchedule<float>();
            
var pipelineModel = new PipelineParallelModel<float, Tensor<float>, Tensor<float>>(
    wrappedModel, config, microBatchCount: 4, schedule: schedule);

Constructors

Constructor Summary
PipelineParallelModel(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config, int microBatchCount, IPipelinePartitionStrategy<T> partitionStrategy, IPipelineSchedule<T> schedule, ActivationCheckpointConfig checkpointConfig) Creates a new Pipeline Parallel model.

Properties

Property Summary
T EstimatedBubbleFraction { get; } Gets the estimated pipeline bubble fraction for the current configuration.

Methods

Method Summary
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters)
ModelMetadata<T> GetModelMetadata()
TOutput Predict(TInput input)
void LoadModel(string filePath)
void SaveModel(string filePath)
void Train(TInput input, TOutput expectedOutput)