FedGENDistillation<T>

Models & Types · AiDotNet.FederatedLearning.Distillation

FedGEN — Data-free federated distillation using a lightweight generator on the server.

Inheritance: FederatedLearningComponentBase<T> → FedGENDistillation<T>

Implements: IFederatedDistillationStrategy<T>

For Beginners

Instead of needing a shared dataset, the server learns a tiny “imagination” network that produces feature vectors every client’s classifier agrees on. Clients then practise on these imagined features, which transfers what the other clients know without sharing any real data.

How It Works

FedGEN (Zhu et al., 2021) learns a small conditional generator G_w(y, ε) on the server that maps a label and Gaussian noise to the latent feature space of the clients’ predictors. The generator is trained so that the label-count-weighted ensemble of client predictor heads classifies its samples as the requested label: min_w E_{y ~ p̂(y), ε ~ N(0, I)} CE(Σ_k α_k^y h_k(G_w(y, ε)), y), with α_k^y = n_k^y / Σ_j n_j^y. Clients then distill this ensemble knowledge into their own head with the data-free term E_{ŷ ~ p̂(y), z ~ G_w(·|ŷ)} CE(h(z; θ_p), ŷ).

Model layout: the first C·L entries of a client’s parameter vector are its predictor weights W [C, L] (row-major) and the next C entries its bias b, where C is the class count and L the latent dimension; the remaining entries (the feature extractor) are not touched by distillation. The generator is a two-layer MLP [onehot(y); ε] → ReLU(hidden) → z, sized by the constructor options.

Generator objective: following the paper and the authors’ released code (serverpFedGen.train_generator), each server step minimizes TeacherLossWeight · L_teacher − StudentLossWeight · KL + DiversityLossWeight · L_div (ensemble_alpha = 1, ensemble_beta = 0 and ensemble_eta = 1 in the official configs). L_teacher is selected by TeacherLoss; KL = KL(softmax(Σ_k α_k^y h_k(z)) ‖ softmax(h_s(z))) averaged over every entry, with h_s the sample-count-weighted average head (the server model); and L_div = exp(−mean_{i,j} ‖ε_i − ε_j‖²/d_ε · ‖z_i − z_j‖₁/L).

Generator optimizer: as in the official code, the generator is trained with Adam (lr 3e-4, betas (0.9, 0.999), eps 1e-8, L2 weight decay 1e-2 added to the gradient) and the learning rate decays by 0.98 after every round (ExponentialLR). GeneratorOptimizer and the related properties change this.

Knowledge exchange:ExtractKnowledge returns the head as a [C, L + 5] matrix (W | b | per-class sample count as four base-256 digits, least significant first). The digits are exact in every numeric type (BFloat16 holds integers exactly only up to 256 and Half up to 2048), so counts of any size reach the server unchanged. AggregateKnowledge trains the generator against the ensemble of these heads and returns a single-row matrix holding the generator parameters followed by the global label distribution p̂(y), which ApplyKnowledge uses on the client.

Reference: Zhu et al. (2021), “Data-Free Knowledge Distillation for Heterogeneous Federated Learning”.

Constructors

Constructor Summary
FedGENDistillation(int numClasses, int featureDim, int generatorEpochs, double generatorLearningRate, int seed, int noiseDim, int generatorHiddenDim, int batchSize, int distillationEpochs, double distillationLearningRate) Creates a new FedGEN distillation strategy.

Properties

Property Summary
FedGenGeneratorOptimizer GeneratorOptimizer { get; set; } Update rule of the server generator.
FedGenTeacherLoss TeacherLoss { get; set; } Form of the teacher term.
double AdamBeta1 { get; set; } Adam first-moment decay β1.
double AdamBeta2 { get; set; } Adam second-moment decay β2.
double AdamEpsilon { get; set; } Adam denominator term ε.
double DiversityLossWeight { get; set; } Weight of the diversity term that keeps different noise draws from collapsing onto one feature (ensemble_eta in the official code).
double GeneratorLearningRateDecay { get; set; } Factor applied to the generator learning rate after every round (ExponentialLR gamma in the official code).
double GeneratorWeightDecay { get; set; } L2 weight decay added to the generator gradient (weight_decay in the official configs).
double StudentLossWeight { get; set; } Weight of the adversarial student term: the generator maximizes the KL divergence between the teacher ensemble and the server model on its samples (ensemble_beta in the official code).
double TeacherLossWeight { get; set; } Weight of the teacher term in the generator objective (ensemble_alpha in the official code).
int GeneratorParameterCount { get; } Number of generator parameters.
int HeadParameterCount { get; } Number of client parameters that form the predictor head (C·L weights plus C biases).

Methods

Method Summary
Matrix<T> AggregateKnowledge(Dictionary<int, Matrix<T>> clientKnowledge, Dictionary<int, double> clientWeights)
Matrix<T> ExtractHeadKnowledge(Vector<T> localModelParameters, int[] labelCounts) Extracts the client’s predictor head together with its per-class sample counts.
Matrix<T> ExtractKnowledge(Vector<T> localModelParameters, Matrix<T> publicData)
Matrix<T> GenerateFeatures(int label, int count) Draws latent features from the current server generator.
Vector<T> ApplyKnowledge(Vector<T> localModelParameters, Matrix<T> globalKnowledge, Matrix<T> publicData, double temperature)