Stage3ShardedLinear<T>
Layers · AiDotNet.DistributedTraining.Layers
A linear layer with true ZeRO Stage-3 / FSDP parameter RESIDENCY (Rajbhandari et al. 2020; Zhao
et al. 2023). Each rank keeps ONLY its 1/worldSize shard of the flat weight resident; the
full weight is materialized transiently by an AllGather at the start of Forward (via
FsdpAllGatherParameter) and released as soon as the layer’s matmul is done. The
gradient of the full weight is reduce-scattered back to the shard in the tape backward, so at rest
each rank stores ~fullParams/worldSize + outputSize — peak parameter residency scales with
the layer, not the model. This is the residency the cache-eviction CpuOffloadParams path
only approximated.
Inheritance: LayerBase<T> → Stage3ShardedLinear<T>
Implements: IShapeContract
Constructors
| Constructor | Summary |
|---|---|
Stage3ShardedLinear(ICommunicationBackend<T> backend, int inputSize, int outputSize, IActivationFunction<T> activationFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ShardLength { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
|
void SetFromFullWeights(Tensor<T> fullWeight, Tensor<T> fullBias) |
Sets this rank’s shard from the FULL weight (row-major flat slice) + the replicated bias; used to build a Stage-3 layer from an unsharded reference in equivalence tests. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |