DirectPreferenceOptimization<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Direct Preference Optimization (DPO) for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → DirectPreferenceOptimization<T, TInput, TOutput>

For Beginners

DPO learns from pairs of responses where one is preferred over the other. Instead of training a reward model first (like RLHF), DPO directly adjusts the model to make preferred responses more likely and rejected responses less likely.

How It Works

DPO is a popular preference optimization method that directly optimizes the model using preference pairs without requiring a separate reward model.

The DPO loss function is: L_DPO = -E[log σ(β * (log π(y_w|x)/π_ref(y_w|x) - log π(y_l|x)/π_ref(y_l|x)))] where y_w is the chosen response, y_l is the rejected response, and β controls the strength of preference learning.

Original paper: “Direct Preference Optimization: Your Language Model is Secretly a Reward Model” by Rafailov et al. (2023)

Constructors

Constructor Summary
DirectPreferenceOptimization(FineTuningOptions<T> options) Initializes a new instance of DPO fine-tuning.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)