FinancialPPOAgent<T>

Models & Types · AiDotNet.Finance.Trading.Agents

Proximal Policy Optimization (Schulman et al. 2017) for trading: the clipped surrogate objective trained over a fixed-length rollout, as the paper and its reference code (OpenAI Baselines ppo1) define it.

Inheritance: ReinforcementLearningAgentBase<T> → TradingAgentBase<T> → FinancialPPOAgent<T>

Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>, IMaskableAgent<T>

For Beginners

PPO improves its trading policy in small, safe steps: it collects a batch of experience, then updates the policy several times on that batch, but stops any action’s probability from moving too far from where it was when the batch was collected. That keeps one unlucky batch from wrecking a policy that was working.

How It Works

The algorithm (paper, Algorithm 1 and §5): act for T steps (RolloutLength, across episode boundaries), estimate advantages with GAE(λ), then optimize L = L^CLIP − c₂·S + c₁·L^VF for K epochs of minibatch Adam over that rollout. L^CLIP = E[min(r·Â, clip(r, 1 − ε, 1 + ε)·Â)] with r = π(a|s) / π_old(a|s).

Defaults are the paper’s MuJoCo setting (Table 3), the one for separate fully-connected policy and value networks, which this agent builds: T = 2048, Adam step size 3e-4, 10 epochs, minibatch 64, γ = 0.99, λ = 0.95, ε = 0.2, no entropy bonus. Baselines’ ppo1 fills what the paper leaves open: two 64-unit tanh layers with normc initialization (0.01 on the policy head), a Gaussian policy with a learned state-independent log standard deviation for continuous actions, observations standardized by running statistics (updated once per rollout) and clipped to ±5, advantages standardized over the rollout, one Adam (ε = 1e-5) over both networks, and the value loss added at weight 1. CleanRL’s additions are options: an entropy bonus (EntropyCoefficient) and global gradient-norm clipping (MaxGradientNorm).

Example

var options = new FinancialPPOAgentOptions<double> { StateSize = 30, ActionSize = 3 };
var agent = new FinancialPPOAgent<double>(
    new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 3),
    new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 1),
    options);

Constructors

Constructor Summary
FinancialPPOAgent(NeuralNetworkArchitecture<T> actorArchitecture, NeuralNetworkArchitecture<T> criticArchitecture, TradingAgentOptions<T> options) Creates a PPO agent from its policy (actor) and value (critic) architectures.

Properties

Property Summary
int FeatureCount { get; }
int RolloutCount { get; } The number of steps collected since the last update.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
ModelOptions GetOptions()
T Train()
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) The policy network’s gradients of lossFunction on one normalized observation.
Vector<T> SelectAction(Vector<T> state, bool training)
Vector<T> SelectAction(Vector<T> state, bool training, bool[] legalActions)
byte[] Serialize() Auto-generated common model serialization surface.
void ApplyGradients(Vector<T> gradients, T learningRate) Applies gradients to the policy network.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done, bool[] nextLegalActions)