SimplePreferenceOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Simple Preference Optimization (SimPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → SimplePreferenceOptimization<T, TInput, TOutput>
For Beginners
SimPO is like DPO but simpler and more memory efficient. It doesn’t need to keep a frozen copy of the original model, which saves GPU memory. Instead, it uses the average log probability of responses and a target reward margin.
How It Works
SimPO is a reference-free preference optimization method that outperforms DPO while being more computationally efficient (no need for a reference model).
Key differences from DPO:
- Uses average log probability instead of sum (length-normalized)
- No reference model needed
- Adds a target reward margin (gamma) for stability
The SimPO loss function is: L_SimPO = -log σ(β/|y| * (log π(y_w|x) - log π(y_l|x)) - γ) where γ is the target margin and |y| denotes response length for normalization.
Original paper: “SimPO: Simple Preference Optimization with a Reference-Free Reward” by Meng et al. (2024) - NeurIPS 2024
Constructors
| Constructor | Summary |
|---|---|
SimplePreferenceOptimization(FineTuningOptions<T> options) |
Initializes a new instance of SimPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |