HybridShardedModel<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements 3D Parallelism (Hybrid Sharded) model - combines data, tensor, and pipeline parallelism.
Inheritance: ShardedModelBase<T, TInput, TOutput> → HybridShardedModel<T, TInput, TOutput>
For Beginners
3D Parallelism is the ultimate distributed training strategy - it combines ALL the techniques:
Imagine training a MASSIVE model across 512 GPUs:
- Pipeline Parallel (depth): Split model into 8 stages (64 GPUs per stage)
- Tensor Parallel (width): Within each stage, split layers 8 ways (8 GPUs per tensor group)
- Data Parallel (batches): Remaining 8 GPUs in each tensor group process different data
Layout example for 512 GPUs = 8 pipeline × 8 tensor × 8 data:
- Stage 0: GPUs 0-63 (layers 0-12)
- Tensor group 0: GPUs 0-7 (data replicas)
- Tensor group 1: GPUs 8-15 (data replicas)
- … 8 tensor groups total
- Stage 1: GPUs 64-127 (layers 13-25)
- …and so on
How It Works
Strategy Overview: 3D Parallelism combines all three major parallelism strategies for maximum scalability:
- Data Parallelism: Different data batches across replicas
- Tensor Parallelism: Layer-wise partitioning within each pipeline stage
- Pipeline Parallelism: Model depth partitioning across stages
This enables training extremely large models (100B+ parameters) on thousands of GPUs by exploiting parallelism in all dimensions. This is the strategy used for training models like GPT-3, Megatron-Turing NLG, and other frontier models.
Use Cases:
- Training frontier models (GPT-3 scale: 100B-1T parameters)
- Requires 100s to 1000s of GPUs
- When single parallelism dimension isn’t enough
- Production training at largest scales (OpenAI, Google, Meta)
Trade-offs:
- Memory: Excellent - exploits all memory-saving strategies
- Communication: Complex - requires careful network topology optimization
- Complexity: Very High - most complex distributed strategy
- Best for: Frontier-scale models (100B+ params), massive GPU clusters
- Requires: Careful tuning of all three parallelism dimensions for efficiency
Implementation Note: This is a production-ready framework providing the 3D parallelism infrastructure. Full production deployment requires:
- Process group management (separate groups for data/tensor/pipeline)
- Model-specific layer partitioning
- Careful configuration tuning for your specific cluster topology
This implementation demonstrates the pattern and provides the foundation.
Example:
Constructors
| Constructor | Summary |
|---|---|
HybridShardedModel(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config, int pipelineParallelSize, int tensorParallelSize, int dataParallelSize) |
Creates a new 3D Parallel (Hybrid Sharded) model. |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters) |
|
ModelMetadata<T> GetModelMetadata() |
|
TOutput Predict(TInput input) |
|
Vector<T> GatherFullParameters() |
Gathers the full parameter vector for the 3D-parallel layout, DE-DUPLICATING data-parallel replicas. |
void LoadModel(string filePath) |
|
void SaveModel(string filePath) |
|
void SynchronizeGradients() |
|
void Train(TInput input, TOutput expectedOutput) |