RowParallelLinear<T>
Layers · AiDotNet.DistributedTraining.Layers
Megatron-LM row-parallel linear layer (Shoeybi et al. 2019, “Megatron-LM”, §3). With the
framework convention Y = X · Wᵀ and W = [outputSize, inputSize], the INPUT dimension
is partitioned across the tensor-parallel ranks: rank r owns the contiguous input slice
W_r = W[:, start_r : start_r+localIn] and consumes the matching input slice
X_r = [batch, localIn] (typically the split output of a preceding
ColumnParallelLinear). Each rank produces a PARTIAL sum Y_r = X_r · W_rᵀ;
the ḡ conjugate operator (ReduceFromTensorParallelRegion) all-reduces the
partials into the full output Y = Σ_r Y_r in the forward pass (identity backward). The bias
is replicated and added ONCE after the reduce. Weight-shard gradients come from the tape (no
manual backward).
Inheritance: LayerBase<T> → RowParallelLinear<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
RowParallelLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, IActivationFunction<T> activationFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int LocalInputSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) |
Sets this rank’s shard from the FULL weight/bias by slicing its input columns; the bias is replicated. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |