RankResponsesHumanFeedback<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Rank Responses to Align Human Feedback (RRHF) for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → RankResponsesHumanFeedback<T, TInput, TOutput>

For Beginners

RRHF learns from a full ranking of responses, not just pairs. If you have 5 responses ranked from best to worst, RRHF uses all that ranking information to train the model more efficiently.

How It Works

RRHF aligns language models by ranking multiple responses and learning from the ranking order rather than just pairwise preferences.

Key features:

  • Uses full ranking information, not just pairs
  • More efficient use of human feedback
  • Combines SFT and ranking objectives

Original paper: “RRHF: Rank Responses to Align Language Models with Human Feedback without tears” by Yuan et al. (2023)

Constructors

Constructor Summary
RankResponsesHumanFeedback(FineTuningOptions<T> options) Initializes a new instance of RRHF fine-tuning.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)