PPOOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for Proximal Policy Optimization (PPO) agents.
Inheritance: ModelOptions → PPOOptions<T>
For Beginners
PPO learns a policy (strategy for choosing actions) by making careful, controlled updates. It’s like learning to drive - you make small adjustments to your steering rather than jerking the wheel wildly. This makes learning stable and efficient.
Key features:
- Actor-Critic: Learns both a policy (actor) and value function (critic)
- Clipped Updates: Prevents too-large changes that could break learning
- GAE: Generalized Advantage Estimation for better gradient estimates
- Multi-Epoch: Reuses collected experience multiple times
Famous for: OpenAI’s ChatGPT uses PPO for RLHF (Reinforcement Learning from Human Feedback)
How It Works
PPO is a state-of-the-art policy gradient algorithm that achieves a balance between sample efficiency, simplicity, and reliability. It uses a clipped surrogate objective to prevent destructively large policy updates.
Constructors
| Constructor | Summary |
|---|---|
PPOOptions() |
Creates options with the default values. |
Properties
| Property | Summary |
|---|---|
ILossFunction<T> ValueLossFunction { get; set; } |
Loss function for value network (typically MSE). |
List<int> PolicyHiddenLayers { get; set; } |
Hidden layer sizes for policy network. |
List<int> ValueHiddenLayers { get; set; } |
Hidden layer sizes for value network. |
T ClipEpsilon { get; set; } |
PPO clipping parameter (epsilon). |
T DiscountFactor { get; set; } |
Discount factor (gamma) for future rewards. |
T EntropyCoefficient { get; set; } |
Entropy coefficient for exploration. |
T GaeLambda { get; set; } |
GAE (Generalized Advantage Estimation) lambda parameter. |
T PolicyLearningRate { get; set; } |
Learning rate for the policy network. |
T ValueLearningRate { get; set; } |
Learning rate for the value network. |
bool IsContinuous { get; set; } |
Whether the action space is continuous (true) or discrete (false). |
double MaxGradNorm { get; set; } |
Maximum gradient norm for gradient clipping. |
int ActionSize { get; set; } |
Number of possible actions (discrete) or action dimensions (continuous). |
int MiniBatchSize { get; set; } |
Mini-batch size for training. |
int StateSize { get; set; } |
Size of the state observation space. |
int StepsPerUpdate { get; set; } |
Number of steps to collect before each training update. |
int TrainingEpochs { get; set; } |
Number of epochs to train on collected data. |