TD3Agent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.TD3

Twin Delayed Deep Deterministic Policy Gradient (TD3) agent for continuous control.

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → TD3Agent<T>

Implements: IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

TD3 is one of the best algorithms for continuous control tasks (like robot movement). It’s more stable and robust than DDPG.

Key innovations:

  • Twin Critics: Uses two Q-networks and takes the minimum to avoid overoptimism
  • Delayed Updates: Waits before updating the policy to let Q-values stabilize
  • Target Smoothing: Adds noise to target actions to prevent exploitation of errors

Think of it like getting a second opinion before making decisions, and taking time to verify information before acting on it.

Used by: Robotic control, autonomous systems, continuous optimization

How It Works

TD3 improves upon DDPG with three key innovations:

  1. Twin Q-Networks: Uses two Q-functions to reduce overestimation bias
  2. Delayed Policy Updates: Updates policy less frequently than Q-networks
  3. Target Policy Smoothing: Adds noise to target actions for robustness

Example

// Create a TD3 agent for stable continuous control
var options = new TD3Options<double> { StateSize = 4, ActionSize = 2};
var agent = new TD3Agent<double>(options);
            
// Select a continuous action with target policy smoothing
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);

Constructors

Constructor Summary
TD3Agent() Initializes a new instance with default settings.
TD3Agent(TD3Options<T> options)

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train()
Task TrainAsync()
Task<Vector<T>> PredictAsync(Vector<T> input)
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction)
Vector<T> Predict(Vector<T> input)
Vector<T> SelectAction(Vector<T> state, bool training)
byte[] Serialize() Auto-generated common model serialization surface.
void ApplyGradients(Vector<T> gradients, T learningRate)
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void ResetEpisode()
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)
void Train(Vector<T> state, Vector<T> target) Performs a one-shot supervised update for the training/test harness.