RowParallelLinear<T>

Layers · AiDotNet.DistributedTraining.Layers

Megatron-LM row-parallel linear layer (Shoeybi et al. 2019, “Megatron-LM”, §3). With the framework convention Y = X · Wᵀ and W = [outputSize, inputSize], the INPUT dimension is partitioned across the tensor-parallel ranks: rank r owns the contiguous input slice W_r = W[:, start_r : start_r+localIn] and consumes the matching input slice X_r = [batch, localIn] (typically the split output of a preceding ColumnParallelLinear). Each rank produces a PARTIAL sum Y_r = X_r · W_rᵀ; the ḡ conjugate operator (ReduceFromTensorParallelRegion) all-reduces the partials into the full output Y = Σ_r Y_r in the forward pass (identity backward). The bias is replicated and added ONCE after the reduce. Weight-shard gradients come from the tape (no manual backward).

Inheritance: LayerBase<T> → RowParallelLinear<T>

Implements: IShapeContract

Constructors

Constructor Summary
RowParallelLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, IActivationFunction<T> activationFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int LocalInputSize { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) Sets this rank’s shard from the FULL weight/bias by slicing its input columns; the bias is replicated.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).