FederatedAdapterTuning<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Implements FedAdapter — federated bottleneck adapter tuning where small adapter modules are inserted into each transformer block and only these are communicated.

Inheritance: FederatedLearningComponentBase<T> → FederatedAdapterTuning<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

Instead of modifying a large model’s weights directly, adapters insert small “bottleneck” layers (down-project → activation → up-project) after the attention and feed-forward layers in each transformer block. Only these tiny bottleneck layers are trained and shared in federated learning, keeping the base model frozen.

How It Works

Architecture per adapter (Houlsby et al. 2019; two adapters per transformer layer, one after the attention sub-layer and one after the feed-forward sub-layer):

Reference: Cai, X., et al. (2023). “FedAdapter: Efficient Federated Learning via Bottleneck Adapters.” NeurIPS Workshop 2023.

Constructors

Constructor Summary
FederatedAdapterTuning(int modelDim, int bottleneckDim, int numAdaptedLayers, int layerDim) Creates a new FedAdapter strategy.

Properties

Property Summary
double CompressionRatio { get; }
int AdapterParameterCount { get; }
int BottleneckDimension { get; } Gets the bottleneck hidden dimension.
int LayerDimension { get; } Gets the transformer hidden dimension.
int NumAdaptedLayers { get; } Gets the number of adapted transformer layers.

Methods

Method Summary
T[] ApplyAdapterForward(T[] input, T[] adapterParams, double residualScale) Applies the adapter forward pass with residual connection: output = x + scale * (UpProject(ReLU(DownProject(x) + b_down)) + b_up).
T[] GetLayerAdapterParams(Vector<T> allAdapterParams, int layerIndex, int position) Extracts adapter parameters for a specific layer and position (attention or FFN).
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)