TensorParallelModel<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements Tensor Parallel model wrapper - splits individual layers across ranks (Megatron-LM style).
Inheritance: ShardedModelBase<T, TInput, TOutput> → TensorParallelModel<T, TInput, TOutput>
For Beginners
Tensor parallelism is like splitting a single large calculation across multiple workers. Imagine a huge spreadsheet calculation - instead of one person doing all the math, we divide the spreadsheet columns across multiple people, each computing their portion simultaneously.
For example, in a neural network layer with a 10000x10000 weight matrix:
- GPU 0 handles columns 0-2499
- GPU 1 handles columns 2500-4999
- GPU 2 handles columns 5000-7499
- GPU 3 handles columns 7500-9999
They compute in parallel, then combine results.
How It Works
Strategy Overview: Tensor Parallelism (Megatron-LM style) partitions individual layers horizontally across processes. For example, a large matrix multiplication is split so each GPU computes only a portion of the output, then results are combined. This is particularly effective for transformer models where attention and feed-forward layers can be partitioned along specific dimensions (column-parallel and row-parallel).
Use Cases:
- Very wide models (large hidden dimensions)
- Transformer models (BERT, GPT) with large attention/FFN layers
- When individual layers are too large for single GPU
- Often combined with pipeline parallelism for maximum scalability
Trade-offs:
- Memory: Excellent for wide layers - each rank stores only portion of weights
- Communication: High - requires AllReduce or AllGather within each layer
- Complexity: Very High - requires model-aware partitioning, specific to layer types
- Best for: Transformer models, very wide layers, fast interconnects (NVLink)
- Limitation: Requires fast communication (high overhead on slow networks)
Implementation Note:
This wrapper is a PARAMETER-REPLICATION fallback for arbitrary (black-box) models: it shards
parameters for storage but reconstructs the full vector for each forward/backward and averages
gradients within the tensor-parallel group (data-parallel semantics over the TP group). It is
NOT compute-partitioned Megatron tensor parallelism. Genuine tensor parallelism — each rank
computing only its slice of every layer with in-layer AllReduce — is provided by
ColumnParallelLinear and
RowParallelLinear (Shoeybi et al. 2019),
whose f/ḡ conjugate operators carry the collectives on the autodiff tape; build models from
those layers for real TP.
⚠️ IMPORTANT LIMITATION - Memory Efficiency: This implementation gathers the full parameter vector on every Train() and Predict() call (via GatherFullParameters and SetParameters), which defeats the memory-saving purpose of true tensor parallelism. While parameters are sharded across ranks for storage, they are reconstructed into the full vector for each forward/backward pass. This means:
- Memory savings are minimal compared to data-parallel training
- Communication overhead is high (AllGather on every forward pass)
- This wrapper primarily provides gradient synchronization, not memory-efficient tensor parallelism
For true memory-efficient tensor parallelism, you would need layer-aware implementations where each rank only loads its parameter shard and performs partial matrix multiplications without ever reconstructing the full parameter vector. This simplified implementation is suitable for:
- Testing and development of distributed training infrastructure
- Scenarios where gradient synchronization is more important than memory efficiency
- Models where memory is not the primary constraint
If memory efficiency is critical, consider using FSDP (Fully Sharded Data Parallel) or ZeRO-3 instead, which shard parameters more aggressively and avoid full parameter reconstruction.
Example:
Constructors
| Constructor | Summary |
|---|---|
TensorParallelModel(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config) |
Creates a new Tensor Parallel model. |
Methods
| Method | Summary |
|---|---|
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters) |
|
ModelMetadata<T> GetModelMetadata() |
|
TOutput Predict(TInput input) |
|
void LoadModel(string filePath) |
|
void SaveModel(string filePath) |
|
void SynchronizeGradients() |
Synchronizes tensor-parallel computation results. |
void Train(TInput input, TOutput expectedOutput) |