SparseLoRA<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Implements SLoRA — Sparse LoRA for communication-efficient federated fine-tuning.

Inheritance: FederatedLearningComponentBase<T> → SparseLoRA<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

Even with LoRA, sending all adapter parameters can be expensive when many layers are adapted. SLoRA adds a sparsity step: after local training, each client identifies which adapter elements changed most and only sends those (top-k by magnitude). The server aggregates the sparse updates and broadcasts the result. This can reduce communication by another 2-10x on top of LoRA’s compression.

How It Works

Algorithm:

Reference: Sparse LoRA for Federated Learning (2024).

Constructors

Constructor Summary
SparseLoRA(int modelDim, int rank, double alpha, double sparsityRatio, int numAdaptedLayers, int layerInputDim, int layerOutputDim) Creates a new SLoRA strategy.

Properties

Property Summary
double CompressionRatio { get; }
double LoraScaling { get; } Gets the LoRA output scaling alpha / rank that the forward pass applies to B*A.
double SparsityRatio { get; } Gets the sparsity ratio (fraction of elements communicated).
int AdapterParameterCount { get; }
int Rank { get; } Gets the LoRA rank.

Methods

Method Summary
SparseUpdate<T> SparsifyDelta(Vector<T> adapterBefore, Vector<T> adapterAfter) Applies top-k sparsification to an adapter update delta, keeping only the largest elements by magnitude.
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> AggregateSparseUpdates(Dictionary<int, SparseUpdate<T>> sparseUpdates, Dictionary<int, double> clientWeights) Aggregates multiple sparse updates from clients.
Vector<T> ApplySparseUpdate(Vector<T> adapter, SparseUpdate<T> sparseUpdate) Applies a sparse update to existing adapter parameters: new = old + sparse_delta.
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)
Vector<T> ToDense(SparseUpdate<T> sparse) Converts a sparse update back to a dense vector (filling non-sparse positions with zero).
int ComputeCommunicationCost(SparseUpdate<T> sparse) Computes the actual communication cost of a sparse update (number of non-zero elements).