OddsRatioPreferenceOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Odds Ratio Preference Optimization (ORPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → OddsRatioPreferenceOptimization<T, TInput, TOutput>
For Beginners
ORPO is a clever method that learns both to produce good outputs (like SFT) and to prefer good outputs over bad ones (like DPO) at the same time. It’s simpler because it doesn’t need a frozen reference model.
How It Works
ORPO combines SFT and preference optimization in a single training objective. It doesn’t require a reference model, making it simpler and more memory efficient.
The ORPO loss combines:
- SFT loss on chosen outputs: -log P(y_w|x)
- Odds ratio loss: -λ * log(odds(y_w) / odds(y_l))
where odds(y) = P(y|x) / (1 - P(y|x))
Original paper: “ORPO: Monolithic Preference Optimization without Reference Model” by Hong et al. (2024)
Constructors
| Constructor | Summary |
|---|---|
OddsRatioPreferenceOptimization(FineTuningOptions<T> options) |
Initializes a new instance of ORPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |