FLoRA<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Implements FLoRA — Federated Low-Rank Adaptation with stacked lossless aggregation.

Inheritance: FederatedLearningComponentBase<T> → FLoRA<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

Standard federated LoRA averages the A and B matrices from different clients, which introduces approximation error. FLoRA instead stacks the local LoRA updates — each client’s (B_k, A_k) pair is concatenated vertically/horizontally, preserving all information. The server then uses an SVD-based compression to bring the stacked result back to the target rank. This gives lossless aggregation without the information loss of simple averaging.

How It Works

Algorithm:

Reference: Wang, Y., et al. (2024). “FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations.” arXiv:2405.14739.

Constructors

Constructor Summary
FLoRA(int modelDim, int rank, double alpha, int numAdaptedLayers, int layerInputDim, int layerOutputDim) Creates a new FLoRA strategy.

Properties

Property Summary
double Alpha { get; } Gets the LoRA alpha scaling factor.
double CompressionRatio { get; }
int AdapterParameterCount { get; }
int Rank { get; } Gets the target LoRA rank.

Methods

Method Summary
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)