DoubleDQNAgent<T>
Models & Types · AiDotNet.ReinforcementLearning.Agents.DoubleDQN
Double Deep Q-Network (Double DQN) agent for reinforcement learning.
Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → DoubleDQNAgent<T>
Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>
For Beginners
Standard DQN tends to overestimate Q-values because it uses the same network to both select and evaluate actions (max operator causes positive bias).
Double DQN fixes this by:
- Using online network to SELECT the best action
- Using target network to EVALUATE that action’s value
Think of it like getting a second opinion: one expert picks what looks best, another expert judges its actual value. This reduces overoptimistic estimates.
Key Improvement: More stable learning, better performance, especially when there’s noise or stochasticity in the environment.
How It Works
Double DQN addresses the overestimation bias in standard DQN by decoupling action selection from action evaluation. It uses the online network to select actions and the target network to evaluate them, leading to more accurate Q-value estimates.
Reference: van Hasselt et al., “Deep Reinforcement Learning with Double Q-learning”, 2015.
Example
// Create a Double DQN agent that reduces Q-value overestimation
var options = new DoubleDQNOptions<double> { StateSize = 4, ActionSize = 2, LearningRate = 0.001 };
var agent = new DoubleDQNAgent<double>(options);
// Select an action for the current state
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);
Constructors
| Constructor | Summary |
|---|---|
DoubleDQNAgent() |
Initializes a new instance with default options. |
DoubleDQNAgent(DoubleDQNOptions<T> options) |
Initializes a new instance of the DoubleDQNAgent class. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
ModelOptions GetOptions() |
|
T Train() |
|
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) |
Computes gradients of the loss with respect to this agent’s parameters, without updating them. |
Vector<T> SelectAction(Vector<T> state, bool training) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void ApplyGradients(Vector<T> gradients, T learningRate) |
Not supported for DoubleDQNAgent. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |