FedMeZO<T>

Models & Types · AiDotNet.FederatedLearning.Adapters

Implements FedMeZO — Memory-efficient Zeroth-Order optimization for federated LLM fine-tuning.

Inheritance: FederatedLearningComponentBase<T> → FedMeZO<T>

Implements: IFederatedAdapterStrategy<T>

For Beginners

Training large language models normally requires storing gradients for all parameters, which takes enormous memory (often 3-4x the model size). Zeroth-order (ZO) optimization estimates gradients by evaluating the loss at two slightly perturbed points — no backpropagation needed. This reduces memory to just the model size plus a random seed. FedMeZO brings this to federated learning: clients only need to share the scalar loss difference and the random seed, making it extremely communication-efficient.

How It Works

ZO gradient estimate (SPSA — Simultaneous Perturbation Stochastic Approximation):

Communication per client: {loss_diff: double, seed: int} — just 12 bytes instead of the full parameter vector (millions to billions of doubles).

Multi-query ZO: For better gradient estimates, multiple perturbation directions can be sampled per step, averaging the estimates.

Reference: Malladi, S., et al. (2024). “Fine-Tuning Language Models with Just Forward Passes.” NeurIPS 2023. FedMeZO extension for federated settings (2024).

Constructors

Constructor Summary
FedMeZO(int modelDim, double perturbationScale, double learningRate, int numPerturbations) Creates a new FedMeZO strategy.

Properties

Property Summary
double CompressionRatio { get; }
double LearningRate { get; } Gets the server learning rate.
double PerturbationScale { get; } Gets the ZO perturbation scale (epsilon).
int AdapterParameterCount { get; }
int NumPerturbations { get; } Gets the number of perturbation directions per step.

Methods

Method Summary
ValueTuple<Vector<T>, Vector<T>> ComputePerturbedWeights(Vector<T> weights, int seed) Computes the perturbed parameter vectors w+ = w + epsilonz and w- = w - epsilonz.
Vector<T> AggregateAdapters(Dictionary<int, Vector<T>> clientAdapters, Dictionary<int, double> clientWeights)
Vector<T> AggregateFromMessages(Dictionary<int, ZOClientMessage[]> clientMessages, Dictionary<int, double> clientWeights) Aggregates ZO gradient estimates from multiple client messages on the server.
Vector<T> EstimateGradient(double lossPlus, double lossMinus, int seed) Estimates the gradient from a single perturbation direction using the loss difference.
Vector<T> EstimateGradientMultiQuery(ValueTuple<double, double, int>[] lossResults) Estimates the gradient using multiple perturbation directions for better accuracy.
Vector<T> ExtractAdapterParameters(Vector<T> fullModelParameters)
Vector<T> GeneratePerturbation(int seed) Generates a random perturbation vector z ~ N(0, I) from a seed.
Vector<T> MergeAdapterParameters(Vector<T> fullModelParameters, Vector<T> aggregatedAdapters)
Vector<T> ReconstructGradientFromMessage(ZOClientMessage message) Reconstructs the gradient estimate on the server from a client’s compact message.
ZOClientMessage CreateClientMessage(double lossPlus, double lossMinus, int seed) Creates the minimal message a client sends to the server in the FedMeZO protocol.