FedKDLosses<T>

Models & Types · AiDotNet.FederatedLearning.Compression

The losses of one FedKD adaptive mutual distillation step, averaged over the batch.

Constructors

Constructor Summary
FedKDLosses(T teacherTaskLoss, T studentTaskLoss, T teacherDistillationLoss, T studentDistillationLoss, T hiddenLoss) Creates the loss breakdown.

Properties

Property Summary
T HiddenLoss { get; } Adaptive hidden-state and attention loss shared by both models.
T StudentDistillationLoss { get; } Adaptive distillation loss of the student, KL(y^t || y^s) / (L_t^t + L_t^s).
T StudentTaskLoss { get; } Student (mentee) cross-entropy task loss.
T StudentTotalLoss { get; } Total student loss L^s = L_t^s + L_d^s + L_h.
T TeacherDistillationLoss { get; } Adaptive distillation loss of the teacher, KL(y^s || y^t) / (L_t^t + L_t^s).
T TeacherTaskLoss { get; } Teacher (mentor) cross-entropy task loss.
T TeacherTotalLoss { get; } Total teacher loss L^t = L_t^t + L_d^t + L_h.