ReinforcementLearningHumanFeedback<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Reinforcement Learning from Human Feedback (RLHF) with PPO for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → ReinforcementLearningHumanFeedback<T, TInput, TOutput>
For Beginners
RLHF is like training a model with a coach (reward model) that tells it how good its responses are. The model learns to generate responses that the coach rates highly, while not straying too far from its original behavior.
How It Works
RLHF is the foundational approach for aligning language models with human preferences. It uses a reward model trained on human feedback to guide policy optimization via PPO.
RLHF pipeline:
- Train a reward model on human preference data
- Use PPO to optimize the policy against the reward model
- Add KL penalty to prevent reward hacking
Original paper: “Training language models to follow instructions with human feedback” by Ouyang et al. (2022) - InstructGPT
Constructors
| Constructor | Summary |
|---|---|
ReinforcementLearningHumanFeedback(FineTuningOptions<T> options) |
Initializes a new instance of RLHF fine-tuning with PPO. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |
|
void SetRewardFunction(Func<TInput, TOutput, double> rewardFunction) |
Sets the reward function (or reward model) for evaluating responses. |
void SetValueModel(IFullModel<T, TInput, TOutput> valueModel) |
Sets the value model for PPO advantage estimation. |