TRPOOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for Trust Region Policy Optimization (TRPO) agents.
Inheritance: ModelOptions → ReinforcementLearningOptions<T> → TRPOOptions<T>
For Beginners
TRPO is like learning carefully - it never makes a change that’s “too big”. By limiting how much the policy can change, it guarantees that performance never gets worse (monotonic improvement).
Key features:
- Trust Region: Limits policy change per update (via KL divergence)
- Monotonic Improvement: Guarantees performance doesn’t degrade
- Conjugate Gradient: Efficiently solves constrained optimization
- Line Search: Ensures constraints are satisfied
Think of it like taking small, safe steps when walking on uncertain terrain rather than making large leaps that might cause you to fall.
Famous for: OpenAI’s robotics research, predecessor to PPO
How It Works
TRPO ensures monotonic improvement by constraining policy updates to a “trust region” using KL divergence. This prevents destructively large updates.
Constructors
| Constructor | Summary |
|---|---|
TRPOOptions() |
Creates options with the default values. |
Properties
| Property | Summary |
|---|---|
ILossFunction<T> ValueLossFunction { get; set; } |
Loss function used to train the value network. |
List<int> PolicyHiddenLayers { get; set; } |
Hidden layer widths of the policy (tanh). |
List<int> ValueHiddenLayers { get; set; } |
Hidden layer widths of the value baseline (tanh). |
T GaeLambda { get; set; } |
GAE lambda of the advantage estimate. |
T MaxKL { get; set; } |
The trust region δ, the largest mean KL divergence of a step. |
T ValueLearningRate { get; set; } |
Adam learning rate of the value baseline. |
bool IsContinuous { get; set; } |
Whether the action space is continuous rather than discrete. |
double Damping { get; set; } |
Damping added to the Fisher-vector products of the conjugate gradient solve. |
double LineSearchAcceptRatio { get; set; } |
The smallest accepted fraction of the surrogate’s linearly predicted improvement. |
double LineSearchBacktrackCoeff { get; set; } |
The line search’s exponential shrink factor (appendix C). |
int ActionSize { get; set; } |
Number of discrete actions, or the dimension of the action vector for a continuous action space. |
int ConjugateGradientIterations { get; set; } |
Conjugate gradient iterations of the search direction (appendix C). |
int LineSearchSteps { get; set; } |
Backtracking steps of the line search. |
int StateSize { get; set; } |
Number of features in one observation of the environment state. |
int StepsPerUpdate { get; set; } |
Number of environment steps collected before each policy update. |
int ValueIterations { get; set; } |
Number of value-function gradient steps per update. |