FLoRA<T>
Models & Types · AiDotNet.FederatedLearning.Adapters
Implements FLoRA — Federated Low-Rank Adaptation with stacked lossless aggregation.
Inheritance: FederatedLearningComponentBase<T> → FLoRA<T>
Implements: IFederatedAdapterStrategy<T>
For Beginners
Standard federated LoRA averages the A and B matrices from different clients, which introduces approximation error. FLoRA instead stacks the local LoRA updates — each client’s (B_k, A_k) pair is concatenated vertically/horizontally, preserving all information. The server then uses an SVD-based compression to bring the stacked result back to the target rank. This gives lossless aggregation without the information loss of simple averaging.
How It Works
Algorithm:
Reference: Wang, Y., et al. (2024). “FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations.” arXiv:2405.14739.
Constructors
| Constructor | Summary |
|---|---|
FLoRA(int modelDim, int rank, double alpha, int numAdaptedLayers, int layerInputDim, int layerOutputDim) |
Creates a new FLoRA strategy. |
Properties
| Property | Summary |
|---|---|
double Alpha { get; } |
Gets the LoRA alpha scaling factor. |
double CompressionRatio { get; } |
|
int AdapterParameterCount { get; } |
|
int Rank { get; } |
Gets the target LoRA rank. |
Methods
| Method | Summary |
|---|---|
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) |
|
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters) |
|
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters) |