OddsRatioPreferenceOptimization<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Odds Ratio Preference Optimization (ORPO) for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → OddsRatioPreferenceOptimization<T, TInput, TOutput>

For Beginners

ORPO is a clever method that learns both to produce good outputs (like SFT) and to prefer good outputs over bad ones (like DPO) at the same time. It’s simpler because it doesn’t need a frozen reference model.

How It Works

ORPO combines SFT and preference optimization in a single training objective. It doesn’t require a reference model, making it simpler and more memory efficient.

The ORPO loss combines:

  1. SFT loss on chosen outputs: -log P(y_w|x)
  2. Odds ratio loss: -λ * log(odds(y_w) / odds(y_l))

where odds(y) = P(y|x) / (1 - P(y|x))

Original paper: “ORPO: Monolithic Preference Optimization without Reference Model” by Hong et al. (2024)

Constructors

Constructor Summary
OddsRatioPreferenceOptimization(FineTuningOptions<T> options) Initializes a new instance of ORPO fine-tuning.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)