TensorParallelTransformerBlock<T>

Layers · AiDotNet.DistributedTraining.Layers

A complete Megatron-style tensor-parallel transformer decoder block for inference/serving (pre-LayerNorm, GPT-style): h + Attn(LN1(h)) then h + MLP(LN2(h)), where the attention is a TensorParallelAttention and the MLP is the column-parallel/row-parallel Megatron pair (ColumnParallelLinear up + activation, RowParallelLinear down). LayerNorms and residuals are replicated on every rank; each attention/MLP runs exactly one all-reduce.

Inheritance: LayerBase<T> → TensorParallelTransformerBlock<T>

Implements: IShapeContract

For Beginners

This is one full “layer” of a transformer, but with its heavy matrix-multiplies split across several GPUs. Stack a few of these and you have a large language model whose weights and compute are shared across your GPUs while producing exactly the same output as a single-GPU model.

How It Works

With head count and world size compatible, the sharded block’s output is numerically identical to the un-sharded block — so a model too large for one GPU can be served by splitting each block across GPUs with no change in results. This is the drop-in serving unit that stacks into a tensor-parallel transformer.

Constructors

Constructor Summary
TensorParallelTransformerBlock(ICommunicationBackend<T> backend, int embedDim, int numHeads, int ffnDim, IActivationFunction<T> activation, bool causal) Creates a tensor-parallel transformer block sharded across the ranks of backend.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState()
void SetFromFullWeights(Tensor<T> qW, Tensor<T> qB, Tensor<T> kW, Tensor<T> kB, Tensor<T> vW, Tensor<T> vB, Tensor<T> oW, Tensor<T> oB, Tensor<T> upW, Tensor<T> upB, Tensor<T> downW, Tensor<T> downB) Seeds the sharded projections (attention Q/K/V/O and MLP up/down) from full un-sharded weights.
void UpdateParameters(T learningRate)