Stage3ShardedLinear<T>

Layers · AiDotNet.DistributedTraining.Layers

A linear layer with true ZeRO Stage-3 / FSDP parameter RESIDENCY (Rajbhandari et al. 2020; Zhao et al. 2023). Each rank keeps ONLY its 1/worldSize shard of the flat weight resident; the full weight is materialized transiently by an AllGather at the start of Forward (via FsdpAllGatherParameter) and released as soon as the layer’s matmul is done. The gradient of the full weight is reduce-scattered back to the shard in the tape backward, so at rest each rank stores ~fullParams/worldSize + outputSize — peak parameter residency scales with the layer, not the model. This is the residency the cache-eviction CpuOffloadParams path only approximated.

Inheritance: LayerBase<T> → Stage3ShardedLinear<T>

Implements: IShapeContract

Constructors

Constructor Summary
Stage3ShardedLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, IActivationFunction<T> activationFunction)

Properties

Property Summary
bool SupportsTraining { get; }
int ShardLength { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) Sets this rank’s shard from the FULL weight (row-major flat slice) + the replicated bias; used to build a Stage-3 layer from an unsharded reference in equivalence tests.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).