ColumnParallelLinear<T>

Layers · AiDotNet.DistributedTraining.Layers

Megatron-LM column-parallel linear layer (Shoeybi et al. 2019, “Megatron-LM”, §3). With the framework convention Y = X · Wᵀ and W = [outputSize, inputSize], the OUTPUT dimension is partitioned across the tensor-parallel ranks: rank r owns the contiguous output slice W_r = W[start_r : start_r+localOut, :] and computes its own output columns Y_r = X · W_rᵀ. There is NO forward communication (the split output feeds a following RowParallelLinear); the input is wrapped in the f conjugate operator (CopyToTensorParallelRegion) whose tape backward all-reduces the input gradient dX = Σ_r dX_r. Weight/bias-shard gradients are produced automatically by the tape from the Engine matmul (no manual backward — the framework is tape-autodiff only).

Inheritance: LayerBase<T> → ColumnParallelLinear<T>

Implements: IShapeContract

Constructors

Constructor Summary
ColumnParallelLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, bool gatherOutput, IActivationFunction<T> activationFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int LocalOutputSize { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) Sets this rank’s shard from the FULL weight/bias by slicing its output rows — used to build a parallel layer from an unsharded reference in equivalence tests.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).