ContrastivePreferenceOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Contrastive Preference Optimization (CPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → ContrastivePreferenceOptimization<T, TInput, TOutput>
For Beginners
CPO trains the model by directly comparing preferred and rejected responses. It’s simpler than DPO because it doesn’t need to keep a frozen copy of the original model.
How It Works
CPO is a variant of DPO that focuses on contrastive learning between chosen and rejected responses without requiring a reference model, similar to ORPO but with a different loss formulation.
The CPO loss directly maximizes the gap between chosen and rejected outputs: L_CPO = -log σ(β * (log π(y_w|x) - log π(y_l|x)))
Key advantages:
- No reference model needed (saves memory)
- Simpler implementation
- Good for scenarios where you want the model to deviate from its initial behavior
Constructors
| Constructor | Summary |
|---|---|
ContrastivePreferenceOptimization(FineTuningOptions<T> options) |
Initializes a new instance of CPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |