TensorParallelAttention<T>
Layers · AiDotNet.DistributedTraining.Layers
Megatron-style tensor-parallel multi-head self-attention for inference/serving: the Q/K/V projections are
ColumnParallelLinear (each rank owns a contiguous group of attention heads), each rank runs
scaled-dot-product attention over ONLY its local heads (no cross-rank communication inside attention), and
the output projection is RowParallelLinear whose all-reduce sums the per-rank head
contributions into the full output.
Inheritance: LayerBase<T> → TensorParallelAttention<T>
Implements: IShapeContract
For Beginners
Attention has many “heads” that each look at the sequence a bit differently. Tensor parallelism hands each GPU a subset of the heads to compute, then adds the pieces back together — so a model with more heads than one GPU can handle still runs, and the answer is the same as if one GPU did it all.
How It Works
This is the attention counterpart to the tensor-parallel MLP pair. With head count divisible by the tensor-parallel world size, the sharded block is numerically transparent: its output equals the un-sharded attention’s, while the Q/K/V/O weight memory and per-token compute are divided across the ranks. Exactly one collective (the output projection’s all-reduce) runs per attention block.
Constructors
| Constructor | Summary |
|---|---|
TensorParallelAttention(ICommunicationBackend<T> backend, int embedDim, int numHeads, bool causal) |
Creates a tensor-parallel attention block sharded across the ranks of backend. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ResetState() |
|
void SetFromFullWeights(Tensor<T> qWeight, Tensor<T> qBias, Tensor<T> kWeight, Tensor<T> kBias, Tensor<T> vWeight, Tensor<T> vBias, Tensor<T> oWeight, Tensor<T> oBias) |
Seeds all four projections from full (un-sharded) weights, slicing each rank’s shard. |
void UpdateParameters(T learningRate) |