FinancialSACAgent<T>

Models & Types · AiDotNet.Finance.Trading.Agents

Soft Actor-Critic for trading, as Haarnoja et al. define it: by default the ICML paper (2018a), with a state-value network and a fixed temperature, or the follow-up (2018b), with target Q-networks and a learned temperature.

Inheritance: ReinforcementLearningAgentBase<T> → TradingAgentBase<T> → FinancialSACAgent<T>

Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

SAC learns a policy that earns reward while staying as random as it can afford to. The randomness keeps it exploring, and the “temperature” sets how much reward it will give up to stay random.

How It Works

The policy is a Gaussian over pre-squash actions whose mean and log standard deviation the actor predicts from the state (log σ clipped to [−20, 2], as both authors’ codes do), squashed into the position limit: a = MaxPositionSize · tanh(u), u = μ(s) + σ(s)·ε. Its log-density carries the change of variables of 2018a Appendix C, computed exactly: log π(a|s) = log N(u; μ, σ) − Σ log(1 − tanh²u) − dim(A)·log MaxPositionSize.

2018a (ValueNetwork, the default), each gradient step in the order of Algorithm 1: V toward min Q(s, a) − α log π(a|s) with a ~ π (eq. 5); both Q-functions toward r + γ(1 − d)V̄(s') (eq. 8); the policy on E[α log π(ã|s) − min Q(s, ã)] with the reparameterized ã (eq. 12); then V̄ ← τV + (1 − τ)V̄. Every loss on the value functions is ½(prediction − target)².

2018b (AutomaticTemperature, Sac2018b): both Q-functions toward r + γ(1 − d)(min Q̄(s', a') − α log π(a'|s')) with a’ ~ π(s’); the policy as above; the temperature on J(α) = E[−α(log π(a|s) + H̄)] (eq. 18) with H̄ = −dim(A); then both target Q-functions track by Polyak averaging.

Hyperparameters are the papers’ Table 1: Adam (Kingma and Ba, ε = 1e-8) at 3e-4 for every network, two hidden layers of 256 ReLU units with Glorot-uniform weights (the authors’ TensorFlow default), minibatches of 256, γ = 0.99, τ = 0.005. The authors’ code differs from its papers in three places, each an option: the 2018a code computes every step from the same parameters (SimultaneousUpdates) and uses Q1 alone in the policy loss, softlearning averages the Q-functions there (PolicyCriticAggregate) and learns log α (TemperatureParameter).

Example

// 2018a (the default); FinancialSACAgentOptions<double>.Sac2018b() gives the learned-temperature version.
var options = new FinancialSACAgentOptions<double> { StateSize = 30, ActionSize = 3, ContinuousActions = true };
var agent = new FinancialSACAgent<double>(
    new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 6),   // 3 means, 3 log standard deviations
    new NeuralNetworkArchitecture<double>(inputFeatures: 33, outputSize: 1),  // Q(s, a)
    options);

Constructors

Constructor Summary
FinancialSACAgent(NeuralNetworkArchitecture<T> actorArchitecture, NeuralNetworkArchitecture<T> criticArchitecture, TradingAgentOptions<T> options, NeuralNetworkArchitecture<T> valueArchitecture) Creates a SAC agent from its policy (actor) and Q-function (critic) architectures.

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetTradingMetrics()
ModelMetadata<T> GetModelMetadata()
ModelOptions GetOptions()
T Train()
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) The policy network’s gradients of lossFunction on one state.
Vector<T> SelectAction(Vector<T> state, bool training)
byte[] Serialize() Auto-generated common model serialization surface.
void ApplyGradients(Vector<T> gradients, T learningRate) Applies gradients to the policy network.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)