ColumnParallelLinear<T>
Layers · AiDotNet.DistributedTraining.Layers
Megatron-LM column-parallel linear layer (Shoeybi et al. 2019, “Megatron-LM”, §3). With the
framework convention Y = X · Wᵀ and W = [outputSize, inputSize], the OUTPUT dimension
is partitioned across the tensor-parallel ranks: rank r owns the contiguous output slice
W_r = W[start_r : start_r+localOut, :] and computes its own output columns
Y_r = X · W_rᵀ. There is NO forward communication (the split output feeds a following
RowParallelLinear); the input is wrapped in the f conjugate operator
(CopyToTensorParallelRegion) whose tape backward all-reduces the input gradient
dX = Σ_r dX_r. Weight/bias-shard gradients are produced automatically by the tape from the
Engine matmul (no manual backward — the framework is tape-autodiff only).
Inheritance: LayerBase<T> → ColumnParallelLinear<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
ColumnParallelLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, bool gatherOutput, IActivationFunction<T> activationFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int LocalOutputSize { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) |
Sets this rank’s shard from the FULL weight/bias by slicing its output rows — used to build a parallel layer from an unsharded reference in equivalence tests. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |