FederatedLoRA<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Federated LoRA — Low-Rank Adaptation for parameter-efficient federated fine-tuning.

Inheritance: FederatedLearningComponentBase<T> → FederatedLoRA<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

Imagine a large model as a huge spreadsheet. Instead of modifying every cell (billions of values), LoRA only learns a small “correction” matrix that captures the most important changes. In federated LoRA, each device only sends this tiny correction instead of the whole spreadsheet — making it practical to collaboratively fine-tune GPT-scale models across phones or hospitals.

How It Works

LoRA (Hu et al., 2021) decomposes weight updates into low-rank matrices: ΔW = BA where B ∈ R^{d×r} and A ∈ R^{r×k} with rank r ≪ min(d,k). In federated settings, only the LoRA matrices are communicated, reducing bandwidth by 100-1000x for large models.

This implementation follows the FedEx-LoRA approach (ACL 2025) which performs exact aggregation by averaging A and B matrices separately with residual error correction.

References: Hu et al. (2021), “LoRA: Low-Rank Adaptation of Large Language Models”. Sun et al. (2025), “FedEx-LoRA: Exact Aggregation for Federated Low-Rank Adaptation” (ACL 2025).

Constructors

Constructor Summary
FederatedLoRA(int modelDim, int rank, double alpha, int numAdaptedLayers, int layerInputDim, int layerOutputDim) Creates a new federated LoRA strategy.

Properties

Property Summary
double CompressionRatio { get; }
int AdapterParameterCount { get; }

Methods

Method Summary
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)