TensorParallelModel<T, TInput, TOutput>

Models & Types · AiDotNet.DistributedTraining

Implements Tensor Parallel model wrapper - splits individual layers across ranks (Megatron-LM style).

Inheritance: ShardedModelBase<T, TInput, TOutput> → TensorParallelModel<T, TInput, TOutput>

For Beginners

Tensor parallelism is like splitting a single large calculation across multiple workers. Imagine a huge spreadsheet calculation - instead of one person doing all the math, we divide the spreadsheet columns across multiple people, each computing their portion simultaneously.

For example, in a neural network layer with a 10000x10000 weight matrix:

  • GPU 0 handles columns 0-2499
  • GPU 1 handles columns 2500-4999
  • GPU 2 handles columns 5000-7499
  • GPU 3 handles columns 7500-9999

They compute in parallel, then combine results.

How It Works

Strategy Overview: Tensor Parallelism (Megatron-LM style) partitions individual layers horizontally across processes. For example, a large matrix multiplication is split so each GPU computes only a portion of the output, then results are combined. This is particularly effective for transformer models where attention and feed-forward layers can be partitioned along specific dimensions (column-parallel and row-parallel).

Use Cases:

  • Very wide models (large hidden dimensions)
  • Transformer models (BERT, GPT) with large attention/FFN layers
  • When individual layers are too large for single GPU
  • Often combined with pipeline parallelism for maximum scalability

Trade-offs:

  • Memory: Excellent for wide layers - each rank stores only portion of weights
  • Communication: High - requires AllReduce or AllGather within each layer
  • Complexity: Very High - requires model-aware partitioning, specific to layer types
  • Best for: Transformer models, very wide layers, fast interconnects (NVLink)
  • Limitation: Requires fast communication (high overhead on slow networks)

Implementation Note: This wrapper is a PARAMETER-REPLICATION fallback for arbitrary (black-box) models: it shards parameters for storage but reconstructs the full vector for each forward/backward and averages gradients within the tensor-parallel group (data-parallel semantics over the TP group). It is NOT compute-partitioned Megatron tensor parallelism. Genuine tensor parallelism — each rank computing only its slice of every layer with in-layer AllReduce — is provided by ColumnParallelLinear and RowParallelLinear (Shoeybi et al. 2019), whose f/ḡ conjugate operators carry the collectives on the autodiff tape; build models from those layers for real TP.

⚠️ IMPORTANT LIMITATION - Memory Efficiency: This implementation gathers the full parameter vector on every Train() and Predict() call (via GatherFullParameters and SetParameters), which defeats the memory-saving purpose of true tensor parallelism. While parameters are sharded across ranks for storage, they are reconstructed into the full vector for each forward/backward pass. This means:

  • Memory savings are minimal compared to data-parallel training
  • Communication overhead is high (AllGather on every forward pass)
  • This wrapper primarily provides gradient synchronization, not memory-efficient tensor parallelism

For true memory-efficient tensor parallelism, you would need layer-aware implementations where each rank only loads its parameter shard and performs partial matrix multiplications without ever reconstructing the full parameter vector. This simplified implementation is suitable for:

  • Testing and development of distributed training infrastructure
  • Scenarios where gradient synchronization is more important than memory efficiency
  • Models where memory is not the primary constraint

If memory efficiency is critical, consider using FSDP (Fully Sharded Data Parallel) or ZeRO-3 instead, which shard parameters more aggressively and avoid full parameter reconstruction.

Example:

Constructors

Constructor Summary
TensorParallelModel(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config) Creates a new Tensor Parallel model.

Methods

Method Summary
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters)
ModelMetadata<T> GetModelMetadata()
TOutput Predict(TInput input)
void LoadModel(string filePath)
void SaveModel(string filePath)
void SynchronizeGradients() Synchronizes tensor-parallel computation results.
void Train(TInput input, TOutput expectedOutput)