RankResponsesHumanFeedback<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Rank Responses to Align Human Feedback (RRHF) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → RankResponsesHumanFeedback<T, TInput, TOutput>
For Beginners
RRHF learns from a full ranking of responses, not just pairs. If you have 5 responses ranked from best to worst, RRHF uses all that ranking information to train the model more efficiently.
How It Works
RRHF aligns language models by ranking multiple responses and learning from the ranking order rather than just pairwise preferences.
Key features:
- Uses full ranking information, not just pairs
- More efficient use of human feedback
- Combines SFT and ranking objectives
Original paper: “RRHF: Rank Responses to Align Language Models with Human Feedback without tears” by Yuan et al. (2023)
Constructors
| Constructor | Summary |
|---|---|
RankResponsesHumanFeedback(FineTuningOptions<T> options) |
Initializes a new instance of RRHF fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |