FederatedDPO<T>
Models & Types · AiDotNet.FederatedLearning.Alignment
Implements Federated DPO (Direct Preference Optimization) for reward-model-free LLM alignment.
Inheritance: FederatedLearningComponentBase<T> → FederatedDPO<T>
For Beginners
DPO is a simpler alternative to RLHF that skips the reward model entirely. Instead of training a separate reward model and then using RL, DPO directly optimizes the LLM to prefer good responses over bad ones using a binary cross-entropy loss on preference pairs. Federated DPO lets each organization keep their preference data private while collaboratively aligning the model.
How It Works
DPO loss per preference pair (w, l):
where w is the preferred response, l is the dispreferred, and beta controls sharpness.
Reference: FedDPO: Federated Direct Preference Optimization (2024).
https://arxiv.org/abs/2404.18567
Constructors
| Constructor | Summary |
|---|---|
FederatedDPO(FederatedDPOOptions options) |
Creates a new Federated DPO instance. |
Properties
| Property | Summary |
|---|---|
FederatedDPOOptions Options { get; } |
Gets the DPO configuration options. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T[]> AggregateModels(Dictionary<int, Dictionary<string, T[]>> clientModels, Dictionary<int, double> clientWeights) |
Aggregates DPO-trained model updates from multiple clients. |
double ComputeDPOLoss(double[] policyChosenLogProbs, double[] policyRejectedLogProbs, double[] referenceChosenLogProbs, double[] referenceRejectedLogProbs) |
Computes the DPO loss for a batch of preference pairs. |
double ComputePreferenceAccuracy(double[] policyChosenLogProbs, double[] policyRejectedLogProbs, double[] referenceChosenLogProbs, double[] referenceRejectedLogProbs) |
Computes the accuracy of the model’s implicit preference on a batch. |
double[] ComputeDPOGradientWeights(double[] policyChosenLogProbs, double[] policyRejectedLogProbs, double[] referenceChosenLogProbs, double[] referenceRejectedLogProbs) |
Computes per-example DPO gradients (as implicit reward margins) for parameter updates. |
double[] ComputeRewardMargins(double[] policyChosenLogProbs, double[] policyRejectedLogProbs, double[] referenceChosenLogProbs, double[] referenceRejectedLogProbs) |
Computes the implicit reward margin for each preference pair. |