HierarchicalFedLoRA<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Implements HierFedLoRA — Hierarchical LoRA aggregation for edge-cloud federated topologies.

Inheritance: FederatedLearningComponentBase<T> → HierarchicalFedLoRA<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

In a hierarchical FL system, devices first aggregate within their local “edge” group (e.g., same hospital, same region), then edges aggregate with the central cloud server. HierFedLoRA applies different LoRA ranks at different levels: edge clients may use very low ranks (e.g., rank 4) for fast local communication, while the edge-to-cloud aggregation uses a higher rank to preserve more information.

How It Works

Topology:

Rank promotion: When edge-level adapters (rank 4) are sent to the cloud, they are promoted to the higher cloud rank (rank 16) by padding with zeros, then the cloud aggregates in the higher-rank space and distributes back to edges.

Reference: Hierarchical LoRA Aggregation for Cross-Silo Federated Learning (2024).

Constructors

Constructor Summary
HierarchicalFedLoRA(int modelDim, int localRank, int globalRank, int numAdaptedLayers, int layerDim) Creates a new HierFedLoRA strategy.

Properties

Property Summary
double CompressionRatio { get; }
int AdapterParameterCount { get; }
int GlobalRank { get; } Gets the global (cloud) LoRA rank.
int LayerDim { get; } Gets the layer dimension.
int LocalRank { get; } Gets the local (edge) LoRA rank.
int NumAdaptedLayers { get; } Gets the number of adapted layers.

Methods

Method Summary
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> AggregateCloud(Dictionary<int, Vector<T>> edgeAdapters, Dictionary<int, double> edgeWeights, bool demoteResult) Performs cloud-level aggregation of edge adapters.
Vector<T> AggregateEdge(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) Performs edge-level aggregation of client adapters using weighted averaging at local rank.
Vector<T> AggregateHierarchical(Dictionary<int, Dictionary<int, Vector<T>>> edgeGroups, Dictionary<int, double> clientWeights, Dictionary<int, double> edgeWeights) Full hierarchical aggregation: first aggregate within each edge group, then aggregate across edges at the cloud level.
Vector<T> DemoteToLocalRank(Vector<T> globalAdapter) Demotes a global-rank adapter back to local rank by truncating the B and A matrices.
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)
Vector<T> PromoteToGlobalRank(Vector<T> localAdapter) Promotes a local-rank adapter to the global (cloud) rank by zero-padding the B and A matrices.