HeterogeneousLoRA<T>
Models & Types · AiDotNet.FederatedLearning.Adapters
Heterogeneous LoRA — supports different LoRA ranks per client with SVD-based aggregation.
Inheritance: FederatedLearningComponentBase<T> → HeterogeneousLoRA<T>
Implements: IFederatedAdapterStrategy<T>
For Beginners
Not all devices are equal — a powerful server can afford rank 64 while a phone can only handle rank 4. Heterogeneous LoRA lets each device choose its own rank, then intelligently combines them at the server using singular value decomposition to extract the most important adaptation directions.
How It Works
HeLoRA (Yue et al., 2024) and FlexLoRA (Bai et al., 2024) allow each client to use a different LoRA rank based on their computational budget. The server aggregates adapters of varying sizes using SVD-based weight redistribution: it reconstructs full-rank deltas, averages them, then re-decomposes into the target rank.
References: Yue et al. (2024), “HeLoRA: Heterogeneous Low-Rank Adapters for Federated Fine-Tuning”. Bai et al. (2024), “FlexLoRA: Stacking-based Heterogeneous LoRA Aggregation” (NeurIPS 2024).
Constructors
| Constructor | Summary |
|---|---|
HeterogeneousLoRA(int modelDim, int layerDim, int targetRank, int maxRank) |
Creates a new heterogeneous LoRA strategy. |
Properties
| Property | Summary |
|---|---|
double CompressionRatio { get; } |
|
int AdapterParameterCount { get; } |
Methods
| Method | Summary |
|---|---|
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) |
|
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters) |
|
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters) |