IdentityPreferenceOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Identity Preference Optimization (IPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → IdentityPreferenceOptimization<T, TInput, TOutput>
For Beginners
IPO is like DPO but more stable. It learns preferences without being too aggressive, which helps prevent the model from becoming overconfident or forgetting how to generate diverse responses.
How It Works
IPO improves upon DPO by addressing overfitting issues through a different loss function that doesn’t push the rejected response probability to zero.
The IPO loss function is: L_IPO = (log(π(y_w|x)/π_ref(y_w|x)) - log(π(y_l|x)/π_ref(y_l|x)) - 1/(2β))² This squared loss form provides better gradients and prevents collapse.
Original paper: “A General Theoretical Paradigm to Understand Learning from Human Preferences” by Azar et al. (2023)
Constructors
| Constructor | Summary |
|---|---|
IdentityPreferenceOptimization(FineTuningOptions<T> options) |
Initializes a new instance of IPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |