FedMeZO<T>
Models & Types · AiDotNet.FederatedLearning.Adapters
Implements FedMeZO — Memory-efficient Zeroth-Order optimization for federated LLM fine-tuning.
Inheritance: FederatedLearningComponentBase<T> → FedMeZO<T>
Implements: IFederatedAdapterStrategy<T>
For Beginners
Training large language models normally requires storing gradients for all parameters, which takes enormous memory (often 3-4x the model size). Zeroth-order (ZO) optimization estimates gradients by evaluating the loss at two slightly perturbed points — no backpropagation needed. This reduces memory to just the model size plus a random seed. FedMeZO brings this to federated learning: clients only need to share the scalar loss difference and the random seed, making it extremely communication-efficient.
How It Works
ZO gradient estimate (SPSA — Simultaneous Perturbation Stochastic Approximation):
Communication per client: {loss_diff: double, seed: int} — just 12 bytes instead of the full parameter vector (millions to billions of doubles).
Multi-query ZO: For better gradient estimates, multiple perturbation directions can be sampled per step, averaging the estimates.
Reference: Malladi, S., et al. (2024). “Fine-Tuning Language Models with Just Forward Passes.” NeurIPS 2023. FedMeZO extension for federated settings (2024).
Constructors
| Constructor | Summary |
|---|---|
FedMeZO(int modelDim, double perturbationScale, double learningRate, int numPerturbations) |
Creates a new FedMeZO strategy. |
Properties
| Property | Summary |
|---|---|
double CompressionRatio { get; } |
|
double LearningRate { get; } |
Gets the server learning rate. |
double PerturbationScale { get; } |
Gets the ZO perturbation scale (epsilon). |
int AdapterParameterCount { get; } |
|
int NumPerturbations { get; } |
Gets the number of perturbation directions per step. |
Methods
| Method | Summary |
|---|---|
ValueTuple<Vector<T>, Vector<T>> ComputePerturbedWeights(Vector<T> weights, int seed) |
Computes the perturbed parameter vectors w+ = w + epsilonz and w- = w - epsilonz. |
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights) |
|
Vector<T> AggregateFromMessages(Dictionary<int, ZOClientMessage[]> clientMessages, Dictionary<int, double> clientWeights) |
Aggregates ZO gradient estimates from multiple client messages on the server. |
Vector<T> EstimateGradient(double lossPlus, double lossMinus, int seed) |
Estimates the gradient from a single perturbation direction using the loss difference. |
Vector<T> EstimateGradientMultiQuery(ValueTuple<double, double, int>[] lossResults) |
Estimates the gradient using multiple perturbation directions for better accuracy. |
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters) |
|
Vector<T> GeneratePerturbation(int seed) |
Generates a random perturbation vector z ~ N(0, I) from a seed. |
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters) |
|
Vector<T> ReconstructGradientFromMessage(ZOClientMessage message) |
Reconstructs the gradient estimate on the server from a client’s compact message. |
ZOClientMessage CreateClientMessage(double lossPlus, double lossMinus, int seed) |
Creates the minimal message a client sends to the server in the FedMeZO protocol. |