TRPOAgent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.TRPO

Trust Region Policy Optimization (TRPO) agent for reinforcement learning (Schulman et al. 2015).

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → TRPOAgent<T>

Implements: IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

TRPO is like learning carefully - it never makes changes that are “too big”. By limiting how much the policy can change (using KL divergence), it guarantees that performance never degrades (monotonic improvement).

Key innovations:

  • Trust Region: Constraints on policy change (KL divergence ≤ δ)
  • Monotonic Improvement: Provable performance guarantees
  • Conjugate Gradient: Efficient solution to constrained optimization
  • Line Search: Ensures constraints are satisfied

Think of it like walking carefully on uncertain terrain - small, safe steps rather than large leaps that might cause you to fall.

Famous for: OpenAI robotics, predecessor to PPO (which simplified TRPO)

How It Works

The single-path algorithm of section 5.1 with the practical step of section 6 and appendix C: it maximizes the surrogate L(θ) = E[π_θ(a|s) / π_θold(a|s) · A(s, a)] subject to the mean KL divergence D_KL(π_θold || π_θ) ≤ δ. The search direction s ≈ F⁻¹g comes from the conjugate gradient algorithm on Fisher-vector products Fv = JᵀMJv (appendix C.1), the step is scaled to β = sqrt(2δ / sᵀFs), and a line search shrinks it exponentially until the surrogate improves and the KL constraint holds.

The policy (appendix D) is a fully connected network from the state to the mean of a Gaussian with a diagonal, state-independent covariance whose log standard deviations are a separate parameter vector; for discrete actions the network’s outputs are the logits of a categorical distribution. Advantages are discounted returns minus a learned value baseline (GaeLambda 1 is the paper’s Monte Carlo estimate).

Example

// Create a TRPO agent with monotonic improvement guarantees
var options = new TRPOOptions<double> { StateSize = 4, ActionSize = 2};
var agent = new TRPOAgent<double>(options);
            
// Select an action within the trust region
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);

Constructors

Constructor Summary
TRPOAgent() Initializes a new instance with default settings.
TRPOAgent(TRPOOptions<T> options)

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train() One TRPO iteration on the collected transitions: fits the value baseline, then takes the trust-region step on the policy.
Task TrainAsync()
Task<Vector<T>> PredictAsync(Vector<T> input)
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) Computes gradients of the loss with respect to this agent’s parameters, without updating them.
Vector<T> Predict(Vector<T> input)
Vector<T> SelectAction(Vector<T> state, bool training)
byte[] Serialize() Auto-generated common model serialization surface.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void ResetEpisode()
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool terminated, bool truncated)