ContrastivePreferenceOptimization<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Contrastive Preference Optimization (CPO) for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → ContrastivePreferenceOptimization<T, TInput, TOutput>

For Beginners

CPO trains the model by directly comparing preferred and rejected responses. It’s simpler than DPO because it doesn’t need to keep a frozen copy of the original model.

How It Works

CPO is a variant of DPO that focuses on contrastive learning between chosen and rejected responses without requiring a reference model, similar to ORPO but with a different loss formulation.

The CPO loss directly maximizes the gap between chosen and rejected outputs: L_CPO = -log σ(β * (log π(y_w|x) - log π(y_l|x)))

Key advantages:

  • No reference model needed (saves memory)
  • Simpler implementation
  • Good for scenarios where you want the model to deviate from its initial behavior

Constructors

Constructor Summary
ContrastivePreferenceOptimization(FineTuningOptions<T> options) Initializes a new instance of CPO fine-tuning.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)