ReinforcementLearningHumanFeedback<T, TInput, TOutput>

Models & Types · AiDotNet.FineTuning

Implements Reinforcement Learning from Human Feedback (RLHF) with PPO for fine-tuning.

Inheritance: FineTuningBase<T, TInput, TOutput> → ReinforcementLearningHumanFeedback<T, TInput, TOutput>

For Beginners

RLHF is like training a model with a coach (reward model) that tells it how good its responses are. The model learns to generate responses that the coach rates highly, while not straying too far from its original behavior.

How It Works

RLHF is the foundational approach for aligning language models with human preferences. It uses a reward model trained on human feedback to guide policy optimization via PPO.

RLHF pipeline:

  1. Train a reward model on human preference data
  2. Use PPO to optimize the policy against the reward model
  3. Add KL penalty to prevent reward hacking

Original paper: “Training language models to follow instructions with human feedback” by Ouyang et al. (2022) - InstructGPT

Constructors

Constructor Summary
ReinforcementLearningHumanFeedback(FineTuningOptions<T> options) Initializes a new instance of RLHF fine-tuning with PPO.

Properties

Property Summary
FineTuningCategory Category { get; }
bool RequiresReferenceModel { get; }
bool RequiresRewardModel { get; }
string MethodName { get; }

Methods

Method Summary
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken)
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken)
void SetRewardFunction(Func<TInput, TOutput, double> rewardFunction) Sets the reward function (or reward model) for evaluating responses.
void SetValueModel(IFullModel<T, TInput, TOutput> valueModel) Sets the value model for PPO advantage estimation.