FinancialPPOAgent<T>
Models & Types · AiDotNet.Finance.Trading.Agents
Proximal Policy Optimization (Schulman et al. 2017) for trading: the clipped surrogate objective trained over a
fixed-length rollout, as the paper and its reference code (OpenAI Baselines ppo1) define it.
Inheritance: ReinforcementLearningAgentBase<T> → TradingAgentBase<T> → FinancialPPOAgent<T>
Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>, IMaskableAgent<T>
For Beginners
PPO improves its trading policy in small, safe steps: it collects a batch of experience, then updates the policy several times on that batch, but stops any action’s probability from moving too far from where it was when the batch was collected. That keeps one unlucky batch from wrecking a policy that was working.
How It Works
The algorithm (paper, Algorithm 1 and §5): act for T steps (RolloutLength,
across episode boundaries), estimate advantages with GAE(λ), then optimize
L = L^CLIP − c₂·S + c₁·L^VF for K epochs of minibatch Adam over that rollout.
L^CLIP = E[min(r·Â, clip(r, 1 − ε, 1 + ε)·Â)] with r = π(a|s) / π_old(a|s).
Defaults are the paper’s MuJoCo setting (Table 3), the one for separate fully-connected policy and value
networks, which this agent builds: T = 2048, Adam step size 3e-4, 10 epochs, minibatch 64, γ = 0.99, λ = 0.95,
ε = 0.2, no entropy bonus. Baselines’ ppo1 fills what the paper leaves open: two 64-unit tanh layers
with normc initialization (0.01 on the policy head), a Gaussian policy with a learned state-independent log
standard deviation for continuous actions, observations standardized by running statistics (updated once per
rollout) and clipped to ±5, advantages standardized over the rollout, one Adam (ε = 1e-5) over both networks, and
the value loss added at weight 1. CleanRL’s additions are options: an entropy bonus
(EntropyCoefficient) and global gradient-norm clipping
(MaxGradientNorm).
Example
var options = new FinancialPPOAgentOptions<double> { StateSize = 30, ActionSize = 3 };
var agent = new FinancialPPOAgent<double>(
new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 3),
new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 1),
options);
Constructors
| Constructor | Summary |
|---|---|
FinancialPPOAgent(NeuralNetworkArchitecture<T> actorArchitecture, NeuralNetworkArchitecture<T> criticArchitecture, TradingAgentOptions<T> options) |
Creates a PPO agent from its policy (actor) and value (critic) architectures. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int RolloutCount { get; } |
The number of steps collected since the last update. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
T Train() |
|
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) |
The policy network’s gradients of lossFunction on one normalized observation. |
Vector<T> SelectAction(Vector<T> state, bool training) |
|
Vector<T> SelectAction(Vector<T> state, bool training, bool[] legalActions) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void ApplyGradients(Vector<T> gradients, T learningRate) |
Applies gradients to the policy network. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done, bool[] nextLegalActions) |