FedKDCompressor<T>

Models & Types · AiDotNet.FederatedLearning.Compression

Implements FedKD — communication-efficient federated learning via adaptive mutual knowledge distillation and dynamic SVD gradient compression.

Inheritance: FederatedLearningComponentBase<T> → FedKDCompressor<T>

For Beginners

Every client keeps a large private “mentor” (teacher) model and trains it together with a small shared “mentee” (student) model; only the mentee’s updates are sent to the server. The two models teach each other (mutual distillation), and the mentee’s gradients are compressed with a truncated SVD before upload and download.

How It Works

Adaptive mutual distillation (per client, for labelled data):

Dynamic gradient approximation: a gradient matrix g ≈ U Σ V keeps the K largest singular values, the fewest whose energy sum_{j <= K} σ_j² / sum_j σ_j² reaches T(t), where the threshold grows linearly from T_start to T_end over the rounds (paper defaults 0.95 and 0.98).

Reference: Wu, C., Wu, F., Lyu, L., Huang, Y., and Xie, X. (2022). “Communication-efficient federated learning via knowledge distillation.” Nature Communications 13, 2032.

Constructors

Constructor Summary
FedKDCompressor(double temperature, double startEnergyThreshold, double endEnergyThreshold) Creates a new FedKD compressor.

Properties

Property Summary
double EndEnergyThreshold { get; } Gets the last-round SVD energy threshold.
double StartEnergyThreshold { get; } Gets the first-round SVD energy threshold.
double Temperature { get; } Gets the distillation temperature.

Methods

Method Summary
FedKDCompressedGradient<T> CompressGradient(Matrix<T> gradient, double energyThreshold) Compresses a gradient matrix g ≈ U Σ V by keeping the fewest singular values whose energy reaches the threshold.
FedKDLosses<T> ComputeMutualDistillationLosses(Tensor<T> teacherLogits, Tensor<T> studentLogits, int[] labels, Tensor<T> teacherHidden, Tensor<T> studentHidden, Tensor<T> hiddenProjection, Tensor<T> teacherAttention, Tensor<T> studentAttention) Computes the adaptive mutual distillation losses of a teacher (mentor) and student (mentee) pair.
Matrix<T> AggregateCompressedGradients(Dictionary<int, FedKDCompressedGradient<T>> clientGradients, Dictionary<int, double> clientWeights) Server step: reconstructs every client’s compressed gradient and returns their weighted average, which the server compresses again (with CompressGradient) before distributing it.
Matrix<T> DecompressGradient(FedKDCompressedGradient<T> compressed) Reconstructs a gradient matrix U Σ V from its compressed factors.
double GetEnergyThreshold(int round, int totalRounds) Gets the dynamic SVD energy threshold T(t) = T_start + (T_end - T_start) * t / totalRounds.