KahnemanTverskyOptimization<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Kahneman-Tversky Optimization (KTO) for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → KahnemanTverskyOptimization<T, TInput, TOutput>

For Beginners

KTO is based on how humans actually make decisions. People feel losses more strongly than equivalent gains (loss aversion). KTO uses this insight to train models - making them more careful about avoiding bad outputs than they are eager to produce good ones.

How It Works

KTO applies prospect theory from behavioral economics to preference learning. Unlike DPO, it doesn’t require paired data - each example is independently labeled as desirable or undesirable.

Key features:

  • Doesn’t require paired preference data
  • Uses loss aversion (undesirable weight typically higher than desirable weight)
  • More sample efficient for imbalanced datasets

Original paper: “KTO: Model Alignment as Prospect Theoretic Optimization” by Ethayarajh et al. (2024)

Constructors

Constructor Summary
KahnemanTverskyOptimization(FineTuningOptions<T> options) Initializes a new instance of KTO fine-tuning.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)