DirectPreferenceOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Direct Preference Optimization (DPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → DirectPreferenceOptimization<T, TInput, TOutput>
For Beginners
DPO learns from pairs of responses where one is preferred over the other. Instead of training a reward model first (like RLHF), DPO directly adjusts the model to make preferred responses more likely and rejected responses less likely.
How It Works
DPO is a popular preference optimization method that directly optimizes the model using preference pairs without requiring a separate reward model.
The DPO loss function is: L_DPO = -E[log σ(β * (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))] where y_w is the chosen response, y_l is the rejected response, and β controls the strength of preference learning.
Original paper: “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” by Rafailov et al. (2023)
Constructors
| Constructor | Summary |
|---|---|
DirectPreferenceOptimization(FineTuningOptions<T> options) |
Initializes a new instance of DPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |