SparseLoRA<T>
Models & Types · AiDotNet.FederatedLearning.Adapters
Implements SLoRA — Sparse LoRA for communication-efficient federated fine-tuning.
Inheritance: FederatedLearningComponentBase<T> → SparseLoRA<T>
Implements: IFederatedAdapterStrategy<T>
For Beginners
Even with LoRA, sending all adapter parameters can be expensive when many layers are adapted. SLoRA adds a sparsity step: after local training, each client identifies which adapter elements changed most and only sends those (top-k by magnitude). The server aggregates the sparse updates and broadcasts the result. This can reduce communication by another 2-10x on top of LoRA’s compression.
How It Works
Algorithm:
Reference: Sparse LoRA for Federated Learning (2024).
Constructors
| Constructor | Summary |
|---|---|
SparseLoRA(int modelDim, int rank, double alpha, double sparsityRatio, int numAdaptedLayers, int layerInputDim, int layerOutputDim) |
Creates a new SLoRA strategy. |
Properties
| Property | Summary |
|---|---|
double CompressionRatio { get; } |
|
double LoraScaling { get; } |
Gets the LoRA output scaling alpha / rank that the forward pass applies to B*A. |
double SparsityRatio { get; } |
Gets the sparsity ratio (fraction of elements communicated). |
int AdapterParameterCount { get; } |
|
int Rank { get; } |
Gets the LoRA rank. |
Methods
| Method | Summary |
|---|---|
SparseUpdate<T> SparsifyDelta(Vector<T> adapterBefore, Vector<T> adapterAfter) |
Applies top-k sparsification to an adapter update delta, keeping only the largest elements by magnitude. |
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) |
|
Vector<T> AggregateSparseUpdates(Dictionary<int, SparseUpdate<T>> sparseUpdates, Dictionary<int, double> clientWeights) |
Aggregates multiple sparse updates from clients. |
Vector<T> ApplySparseUpdate(Vector<T> adapter, SparseUpdate<T> sparseUpdate) |
Applies a sparse update to existing adapter parameters: new = old + sparse_delta. |
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters) |
|
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters) |
|
Vector<T> ToDense(SparseUpdate<T> sparse) |
Converts a sparse update back to a dense vector (filling non-sparse positions with zero). |
int ComputeCommunicationCost(SparseUpdate<T> sparse) |
Computes the actual communication cost of a sparse update (number of non-zero elements). |