TRPOAgent<T>
Models & Types · AiDotNet.ReinforcementLearning.Agents.TRPO
Trust Region Policy Optimization (TRPO) agent for reinforcement learning (Schulman et al. 2015).
Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → TRPOAgent<T>
Implements: IGradientComputable<T, Vector<T>, Vector<T>>
For Beginners
TRPO is like learning carefully - it never makes changes that are “too big”. By limiting how much the policy can change (using KL divergence), it guarantees that performance never degrades (monotonic improvement).
Key innovations:
- Trust Region: Constraints on policy change (KL divergence ≤ δ)
- Monotonic Improvement: Provable performance guarantees
- Conjugate Gradient: Efficient solution to constrained optimization
- Line Search: Ensures constraints are satisfied
Think of it like walking carefully on uncertain terrain - small, safe steps rather than large leaps that might cause you to fall.
Famous for: OpenAI robotics, predecessor to PPO (which simplified TRPO)
How It Works
The single-path algorithm of section 5.1 with the practical step of section 6 and appendix C: it maximizes the
surrogate L(θ) = E[π_θ(a|s) / π_θold(a|s) · A(s, a)] subject to the mean KL divergence
D_KL(π_θold || π_θ) ≤ δ. The search direction s ≈ F⁻¹g comes from the conjugate gradient algorithm on
Fisher-vector products Fv = JᵀMJv (appendix C.1), the step is scaled to β = sqrt(2δ / sᵀFs), and a line
search shrinks it exponentially until the surrogate improves and the KL constraint holds.
The policy (appendix D) is a fully connected network from the state to the mean of a Gaussian with a diagonal,
state-independent covariance whose log standard deviations are a separate parameter vector; for discrete actions the
network’s outputs are the logits of a categorical distribution. Advantages are discounted returns minus a learned
value baseline (GaeLambda 1 is the paper’s Monte Carlo estimate).
Example
// Create a TRPO agent with monotonic improvement guarantees
var options = new TRPOOptions<double> { StateSize = 4, ActionSize = 2};
var agent = new TRPOAgent<double>(options);
// Select an action within the trust region
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);
Constructors
| Constructor | Summary |
|---|---|
TRPOAgent() |
Initializes a new instance with default settings. |
TRPOAgent(TRPOOptions<T> options) |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
T Train() |
One TRPO iteration on the collected transitions: fits the value baseline, then takes the trust-region step on the policy. |
Task TrainAsync() |
|
Task<Vector<T>> PredictAsync(Vector<T> input) |
|
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) |
Computes gradients of the loss with respect to this agent’s parameters, without updating them. |
Vector<T> Predict(Vector<T> input) |
|
Vector<T> SelectAction(Vector<T> state, bool training) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void ResetEpisode() |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool terminated, bool truncated) |