QLambdaAgent<T>
Models & Types ยท AiDotNet.ReinforcementLearning.Agents.EligibilityTraces
Q(lambda) agent that combines Q-learning with eligibility traces for faster credit assignment in tabular reinforcement learning environments.
Inheritance: ReinforcementLearningAgentBase<T> โ QLambdaAgent<T>
For Beginners
In basic Q-learning, only the most recent state-action pair gets updated when a reward arrives. Q(lambda) remembers which states you recently visited and updates all of them, like leaving breadcrumbs that fade over time.
The lambda parameter controls how far back credit is assigned:
- lambda = 0: Same as regular Q-learning (only update last step)
- lambda = 1: Update all visited states equally (Monte Carlo-like)
- lambda = 0.9: Good default, updates recent states more than older ones
This helps the agent learn much faster because rewards propagate backwards through the trajectory in a single episode instead of requiring many episodes.
How It Works
Q(lambda) extends Q-learning by maintaining eligibility traces that assign credit to recently visited state-action pairs. When a reward is received, all eligible state-action pairs are updated proportionally to their trace value, enabling faster learning in long-horizon tasks.
Example
// Create a Q(lambda) agent with eligibility traces for faster learning
var options = new QLambdaOptions<double> { Lambda = 0.9, LearningRate = 0.1, StateSize = 4, ActionSize = 2 };
var agent = new QLambdaAgent<double>(options);
// Select an action for the current state
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);
Constructors
| Constructor | Summary |
|---|---|
QLambdaAgent() |
Initializes a new instance with default settings. |
QLambdaAgent(QLambdaOptions<T> options) |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
T Train() |
|
Task TrainAsync() |
|
Task<Vector<T>> PredictAsync(Vector<T> input) |
|
Vector<T> Predict(Vector<T> input) |
|
Vector<T> SelectAction(Vector<T> state, bool training) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void ResetEpisode() |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |