TensorParallelTransformerBlock<T>
Layers · AiDotNet.DistributedTraining.Layers
A complete Megatron-style tensor-parallel transformer decoder block for inference/serving (pre-LayerNorm,
GPT-style): h + Attn(LN1(h)) then h + MLP(LN2(h)), where the attention is a
TensorParallelAttention and the MLP is the column-parallel/row-parallel Megatron pair
(ColumnParallelLinear up + activation, RowParallelLinear down). LayerNorms
and residuals are replicated on every rank; each attention/MLP runs exactly one all-reduce.
Inheritance: LayerBase<T> → TensorParallelTransformerBlock<T>
Implements: IShapeContract
For Beginners
This is one full “layer” of a transformer, but with its heavy matrix-multiplies split across several GPUs. Stack a few of these and you have a large language model whose weights and compute are shared across your GPUs while producing exactly the same output as a single-GPU model.
How It Works
With head count and world size compatible, the sharded block’s output is numerically identical to the un-sharded block — so a model too large for one GPU can be served by splitting each block across GPUs with no change in results. This is the drop-in serving unit that stacks into a tensor-parallel transformer.
Constructors
| Constructor | Summary |
|---|---|
TensorParallelTransformerBlock(ICommunicationBackend<T> backend, int embedDim, int numHeads, int ffnDim, IActivationFunction<T> activation, bool causal) |
Creates a tensor-parallel transformer block sharded across the ranks of backend. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ResetState() |
|
void SetFromFullWeights(Tensor<T> qW, Tensor<T> qB, Tensor<T> kW, Tensor<T> kB, Tensor<T> vW, Tensor<T> vB, Tensor<T> oW, Tensor<T> oB, Tensor<T> upW, Tensor<T> upB, Tensor<T> downW, Tensor<T> downB) |
Seeds the sharded projections (attention Q/K/V/O and MLP up/down) from full un-sharded weights. |
void UpdateParameters(T learningRate) |