FedKDCompressor<T>
Models & Types · AiDotNet.FederatedLearning.Compression
Implements FedKD — communication-efficient federated learning via adaptive mutual knowledge distillation and dynamic SVD gradient compression.
Inheritance: FederatedLearningComponentBase<T> → FedKDCompressor<T>
For Beginners
Every client keeps a large private “mentor” (teacher) model and trains it together with a small shared “mentee” (student) model; only the mentee’s updates are sent to the server. The two models teach each other (mutual distillation), and the mentee’s gradients are compressed with a truncated SVD before upload and download.
How It Works
Adaptive mutual distillation (per client, for labelled data):
Dynamic gradient approximation: a gradient matrix g ≈ U Σ V keeps the K largest singular values, the fewest whose energy sum_{j <= K} σ_j² / sum_j σ_j² reaches T(t), where the threshold grows linearly from T_start to T_end over the rounds (paper defaults 0.95 and 0.98).
Reference: Wu, C., Wu, F., Lyu, L., Huang, Y., and Xie, X. (2022). “Communication-efficient federated learning via knowledge distillation.” Nature Communications 13, 2032.
Constructors
| Constructor | Summary |
|---|---|
FedKDCompressor(double temperature, double startEnergyThreshold, double endEnergyThreshold) |
Creates a new FedKD compressor. |
Properties
| Property | Summary |
|---|---|
double EndEnergyThreshold { get; } |
Gets the last-round SVD energy threshold. |
double StartEnergyThreshold { get; } |
Gets the first-round SVD energy threshold. |
double Temperature { get; } |
Gets the distillation temperature. |
Methods
| Method | Summary |
|---|---|
FedKDCompressedGradient<T> CompressGradient(Matrix<T> gradient, double energyThreshold) |
Compresses a gradient matrix g ≈ U Σ V by keeping the fewest singular values whose energy reaches the threshold. |
FedKDLosses<T> ComputeMutualDistillationLosses(Tensor<T> teacherLogits, Tensor<T> studentLogits, int[] labels, Tensor<T> teacherHidden, Tensor<T> studentHidden, Tensor<T> hiddenProjection, Tensor<T> teacherAttention, Tensor<T> studentAttention) |
Computes the adaptive mutual distillation losses of a teacher (mentor) and student (mentee) pair. |
Matrix<T> AggregateCompressedGradients(Dictionary<int, FedKDCompressedGradient<T>> clientGradients, Dictionary<int, double> clientWeights) |
Server step: reconstructs every client’s compressed gradient and returns their weighted average, which the server compresses again (with CompressGradient) before distributing it. |
Matrix<T> DecompressGradient(FedKDCompressedGradient<T> compressed) |
Reconstructs a gradient matrix U Σ V from its compressed factors. |
double GetEnergyThreshold(int round, int totalRounds) |
Gets the dynamic SVD energy threshold T(t) = T_start + (T_end - T_start) * t / totalRounds. |