TensorParallelAttention<T>

Layers · AiDotNet.DistributedTraining.Layers

Megatron-style tensor-parallel multi-head self-attention for inference/serving: the Q/K/V projections are ColumnParallelLinear (each rank owns a contiguous group of attention heads), each rank runs scaled-dot-product attention over ONLY its local heads (no cross-rank communication inside attention), and the output projection is RowParallelLinear whose all-reduce sums the per-rank head contributions into the full output.

Inheritance: LayerBase<T> → TensorParallelAttention<T>

Implements: IShapeContract

For Beginners

Attention has many “heads” that each look at the sequence a bit differently. Tensor parallelism hands each GPU a subset of the heads to compute, then adds the pieces back together — so a model with more heads than one GPU can handle still runs, and the answer is the same as if one GPU did it all.

How It Works

This is the attention counterpart to the tensor-parallel MLP pair. With head count divisible by the tensor-parallel world size, the sharded block is numerically transparent: its output equals the un-sharded attention’s, while the Q/K/V/O weight memory and per-token compute are divided across the ranks. Exactly one collective (the output projection’s all-reduce) runs per attention block.

Constructors

Constructor Summary
TensorParallelAttention(ICommunicationBackend<T> backend, int embedDim, int numHeads, bool causal) Creates a tensor-parallel attention block sharded across the ranks of backend.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState()
void SetFromFullWeights(Tensor<T> qWeight, Tensor<T> qBias, Tensor<T> kWeight, Tensor<T> kBias, Tensor<T> vWeight, Tensor<T> vBias, Tensor<T> oWeight, Tensor<T> oBias) Seeds all four projections from full (un-sharded) weights, slicing each rank’s shard.
void UpdateParameters(T learningRate)