TRPOOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for Trust Region Policy Optimization (TRPO) agents.

Inheritance: ModelOptions → ReinforcementLearningOptions<T> → TRPOOptions<T>

For Beginners

TRPO is like learning carefully - it never makes a change that’s “too big”. By limiting how much the policy can change, it guarantees that performance never gets worse (monotonic improvement).

Key features:

  • Trust Region: Limits policy change per update (via KL divergence)
  • Monotonic Improvement: Guarantees performance doesn’t degrade
  • Conjugate Gradient: Efficiently solves constrained optimization
  • Line Search: Ensures constraints are satisfied

Think of it like taking small, safe steps when walking on uncertain terrain rather than making large leaps that might cause you to fall.

Famous for: OpenAI’s robotics research, predecessor to PPO

How It Works

TRPO ensures monotonic improvement by constraining policy updates to a “trust region” using KL divergence. This prevents destructively large updates.

Constructors

Constructor Summary
TRPOOptions() Creates options with the default values.

Properties

Property Summary
ILossFunction<T> ValueLossFunction { get; set; } Loss function used to train the value network.
List<int> PolicyHiddenLayers { get; set; } Hidden layer widths of the policy (tanh).
List<int> ValueHiddenLayers { get; set; } Hidden layer widths of the value baseline (tanh).
T GaeLambda { get; set; } GAE lambda of the advantage estimate.
T MaxKL { get; set; } The trust region δ, the largest mean KL divergence of a step.
T ValueLearningRate { get; set; } Adam learning rate of the value baseline.
bool IsContinuous { get; set; } Whether the action space is continuous rather than discrete.
double Damping { get; set; } Damping added to the Fisher-vector products of the conjugate gradient solve.
double LineSearchAcceptRatio { get; set; } The smallest accepted fraction of the surrogate’s linearly predicted improvement.
double LineSearchBacktrackCoeff { get; set; } The line search’s exponential shrink factor (appendix C).
int ActionSize { get; set; } Number of discrete actions, or the dimension of the action vector for a continuous action space.
int ConjugateGradientIterations { get; set; } Conjugate gradient iterations of the search direction (appendix C).
int LineSearchSteps { get; set; } Backtracking steps of the line search.
int StateSize { get; set; } Number of features in one observation of the environment state.
int StepsPerUpdate { get; set; } Number of environment steps collected before each policy update.
int ValueIterations { get; set; } Number of value-function gradient steps per update.