PPOOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for Proximal Policy Optimization (PPO) agents.

Inheritance: ModelOptions → PPOOptions<T>

For Beginners

PPO learns a policy (strategy for choosing actions) by making careful, controlled updates. It’s like learning to drive - you make small adjustments to your steering rather than jerking the wheel wildly. This makes learning stable and efficient.

Key features:

  • Actor-Critic: Learns both a policy (actor) and value function (critic)
  • Clipped Updates: Prevents too-large changes that could break learning
  • GAE: Generalized Advantage Estimation for better gradient estimates
  • Multi-Epoch: Reuses collected experience multiple times

Famous for: OpenAI’s ChatGPT uses PPO for RLHF (Reinforcement Learning from Human Feedback)

How It Works

PPO is a state-of-the-art policy gradient algorithm that achieves a balance between sample efficiency, simplicity, and reliability. It uses a clipped surrogate objective to prevent destructively large policy updates.

Constructors

Constructor Summary
PPOOptions() Creates options with the default values.

Properties

Property Summary
ILossFunction<T> ValueLossFunction { get; set; } Loss function for value network (typically MSE).
List<int> PolicyHiddenLayers { get; set; } Hidden layer sizes for policy network.
List<int> ValueHiddenLayers { get; set; } Hidden layer sizes for value network.
T ClipEpsilon { get; set; } PPO clipping parameter (epsilon).
T DiscountFactor { get; set; } Discount factor (gamma) for future rewards.
T EntropyCoefficient { get; set; } Entropy coefficient for exploration.
T GaeLambda { get; set; } GAE (Generalized Advantage Estimation) lambda parameter.
T PolicyLearningRate { get; set; } Learning rate for the policy network.
T ValueLearningRate { get; set; } Learning rate for the value network.
bool IsContinuous { get; set; } Whether the action space is continuous (true) or discrete (false).
double MaxGradNorm { get; set; } Maximum gradient norm for gradient clipping.
int ActionSize { get; set; } Number of possible actions (discrete) or action dimensions (continuous).
int MiniBatchSize { get; set; } Mini-batch size for training.
int StateSize { get; set; } Size of the state observation space.
int StepsPerUpdate { get; set; } Number of steps to collect before each training update.
int TrainingEpochs { get; set; } Number of epochs to train on collected data.