FederatedAdapterTuning<T>
Models & Types · AiDotNet.FederatedLearning.Adapters
Implements FedAdapter — federated bottleneck adapter tuning where small adapter modules are inserted into each transformer block and only these are communicated.
Inheritance: FederatedLearningComponentBase<T> → FederatedAdapterTuning<T>
Implements: IFederatedAdapterStrategy<T>
For Beginners
Instead of modifying a large model’s weights directly, adapters insert small “bottleneck” layers (down-project → activation → up-project) after the attention and feed-forward layers in each transformer block. Only these tiny bottleneck layers are trained and shared in federated learning, keeping the base model frozen.
How It Works
Architecture per adapter (Houlsby et al. 2019; two adapters per transformer layer, one after the attention sub-layer and one after the feed-forward sub-layer):
Reference: Cai, X., et al. (2023). “FedAdapter: Efficient Federated Learning via Bottleneck Adapters.” NeurIPS Workshop 2023.
Constructors
| Constructor | Summary |
|---|---|
FederatedAdapterTuning(int modelDim, int bottleneckDim, int numAdaptedLayers, int layerDim) |
Creates a new FedAdapter strategy. |
Properties
| Property | Summary |
|---|---|
double CompressionRatio { get; } |
|
int AdapterParameterCount { get; } |
|
int BottleneckDimension { get; } |
Gets the bottleneck hidden dimension. |
int LayerDimension { get; } |
Gets the transformer hidden dimension. |
int NumAdaptedLayers { get; } |
Gets the number of adapted transformer layers. |
Methods
| Method | Summary |
|---|---|
T[] ApplyAdapterForward(T[] input, T[] adapterParams, double residualScale) |
Applies the adapter forward pass with residual connection: output = x + scale * (UpProject(ReLU(DownProject(x) + b_down)) + b_up). |
T[] GetLayerAdapterParams(Vector<T> allAdapterParams, int layerIndex, int position) |
Extracts adapter parameters for a specific layer and position (attention or FFN). |
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) |
|
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters) |
|
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters) |