FinancialSACAgent<T>
Models & Types · AiDotNet.Finance.Trading.Agents
Soft Actor-Critic for trading, as Haarnoja et al. define it: by default the ICML paper (2018a), with a state-value network and a fixed temperature, or the follow-up (2018b), with target Q-networks and a learned temperature.
Inheritance: ReinforcementLearningAgentBase<T> → TradingAgentBase<T> → FinancialSACAgent<T>
Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>
For Beginners
SAC learns a policy that earns reward while staying as random as it can afford to. The randomness keeps it exploring, and the “temperature” sets how much reward it will give up to stay random.
How It Works
The policy is a Gaussian over pre-squash actions whose mean and log standard deviation the actor predicts
from the state (log σ clipped to [−20, 2], as both authors’ codes do), squashed into the position limit:
a = MaxPositionSize · tanh(u), u = μ(s) + σ(s)·ε. Its log-density carries the change of variables of
2018a Appendix C, computed exactly: log π(a|s) = log N(u; μ, σ) − Σ log(1 − tanh²u) − dim(A)·log MaxPositionSize.
2018a (ValueNetwork, the default), each gradient step in the order of Algorithm 1:
V toward min Q(s, a) − α log π(a|s) with a ~ π (eq. 5); both Q-functions toward r + γ(1 − d)V̄(s')
(eq. 8); the policy on E[α log π(ã|s) − min Q(s, ã)] with the reparameterized ã (eq. 12); then
V̄ ← τV + (1 − τ)V̄. Every loss on the value functions is ½(prediction − target)².
2018b (AutomaticTemperature, Sac2018b):
both Q-functions toward r + γ(1 − d)(min Q̄(s', a') − α log π(a'|s')) with a’ ~ π(s’); the policy as above;
the temperature on J(α) = E[−α(log π(a|s) + H̄)] (eq. 18) with H̄ = −dim(A); then both target Q-functions
track by Polyak averaging.
Hyperparameters are the papers’ Table 1: Adam (Kingma and Ba, ε = 1e-8) at 3e-4 for every network, two hidden
layers of 256 ReLU units with Glorot-uniform weights (the authors’ TensorFlow default), minibatches of 256,
γ = 0.99, τ = 0.005. The authors’ code differs from its papers in three places, each an option: the 2018a code
computes every step from the same parameters (SimultaneousUpdates) and
uses Q1 alone in the policy loss, softlearning averages the Q-functions there
(PolicyCriticAggregate) and learns log α
(TemperatureParameter).
Example
// 2018a (the default); FinancialSACAgentOptions<double>.Sac2018b() gives the learned-temperature version.
var options = new FinancialSACAgentOptions<double> { StateSize = 30, ActionSize = 3, ContinuousActions = true };
var agent = new FinancialSACAgent<double>(
new NeuralNetworkArchitecture<double>(inputFeatures: 30, outputSize: 6), // 3 means, 3 log standard deviations
new NeuralNetworkArchitecture<double>(inputFeatures: 33, outputSize: 1), // Q(s, a)
options);
Constructors
| Constructor | Summary |
|---|---|
FinancialSACAgent(NeuralNetworkArchitecture<T> actorArchitecture, NeuralNetworkArchitecture<T> criticArchitecture, TradingAgentOptions<T> options, NeuralNetworkArchitecture<T> valueArchitecture) |
Creates a SAC agent from its policy (actor) and Q-function (critic) architectures. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetTradingMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
T Train() |
|
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) |
The policy network’s gradients of lossFunction on one state. |
Vector<T> SelectAction(Vector<T> state, bool training) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void ApplyGradients(Vector<T> gradients, T learningRate) |
Applies gradients to the policy network. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |