GroupRelativePolicyOptimization<T, TInput, TOutput>
Models & Types · AiDotNet.FineTuning
Implements Group Relative Policy Optimization (GRPO) for fine-tuning.
Inheritance: FineTuningBase<T, TInput, TOutput> → GroupRelativePolicyOptimization<T, TInput, TOutput>
For Beginners
GRPO is like PPO but more efficient. Instead of training a separate critic model to estimate value, GRPO generates multiple responses for each prompt and uses the group’s average reward as the baseline. This makes training faster and uses less GPU memory.
How It Works
GRPO is a memory-efficient reinforcement learning algorithm developed by DeepSeek that eliminates the need for a separate critic model, reducing memory requirements by ~50%.
Key features of GRPO:
- No critic model needed (saves ~50% memory)
- Group-based advantage estimation
- Works well with verifiable rewards (RLVR)
- Used in DeepSeekMath and DeepSeek-R1
The GRPO advantage is computed as: A_i = (r_i - mean(r_group)) / std(r_group) where r_group are the rewards for all responses in the group.
Original paper: “DeepSeekMath: Pushing the Limits of Mathematical Reasoning” by Shao et al. (2024)
Constructors
| Constructor | Summary |
|---|---|
GroupRelativePolicyOptimization(FineTuningOptions<T> options) |
Initializes a new instance of GRPO fine-tuning. |
Properties
| Property | Summary |
|---|---|
FineTuningCategory Category { get; } |
|
bool RequiresReferenceModel { get; } |
|
bool RequiresRewardModel { get; } |
|
string MethodName { get; } |
Methods
| Method | Summary |
|---|---|
Task<FineTuningMetrics<T>> EvaluateAsync(IFullModel<T, TInput, TOutput> model, FineTuningData<T, TInput, TOutput> evaluationData, CancellationToken cancellationToken) |
|
Task<IFullModel<T, TInput, TOutput>> FineTuneAsync(IFullModel<T, TInput, TOutput> baseModel, FineTuningData<T, TInput, TOutput> trainingData, CancellationToken cancellationToken) |
|
void SetRewardFunction(Func<TInput, TOutput, double> rewardFunction) |
Sets the reward function for evaluating responses. |