DreamerAgent<T>
Models & Types · AiDotNet.ReinforcementLearning.Agents.Dreamer
Dreamer agent (Hafner et al. 2020, “Dream to Control: Learning Behaviors by Latent Imagination”).
Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → DreamerAgent<T>
For Beginners
Dreamer learns a “mental model” of how the environment works, then uses that model to imagine future scenarios and plan actions - like chess players thinking multiple moves ahead.
Key components:
- Representation Model: Combines the agent’s memory with the current observation
- Transition Model: Predicts the next latent state without seeing the observation
- Reward / Discount Models: Predict rewards and whether the episode continues
- Value Model: Estimates how good an imagined state is
- Action Model: Learns the policy entirely in imagination
Advantages: Sample efficient, works with images, enables planning
How It Works
World model (Section 4, Equations 9-10; Appendix A). A recurrent state-space model (RSSM, Hafner et
al. 2018) whose model state s_t = (h_t, z_t) pairs a deterministic GRU state h_t
(DeterministicSize) with a diagonal-Gaussian stochastic state z_t
(StochasticSize):
h_t = GRU(h_{t-1}, [z_{t-1}, a_{t-1}]); transition (prior) p(z_t | h_t); representation (posterior)
q(z_t | h_t, e_t) with e_t = enc(o_t); observation model q(o_t | s_t); reward model q(r_t | s_t); and, for
tasks with early termination, a discount model trained toward the soft labels 0 and gamma (Section 3,
Appendix A). The model maximises Equation 10 on BatchLength-step sequences
drawn from the replay data: reconstruction + RewardScale * reward
ContinueScale* discount log-likelihoods, minusKLScale* KL(posterior || prior) clipped belowFreeNats. (The paper writes the representation model as p and the transition model as q; this class uses the RSSM’s posterior / prior names.)
Behaviour learning (Section 3, Equations 2-8). From every posterior state of the sampled sequences
the agent imagines ImaginationHorizon steps with the transition model and
a tanh-transformed Gaussian action model (Equation 3), computes the V-lambda targets of Equation 6, and
trains the action model by backpropagating V-lambda through the imagined dynamics (Equation 7) and the
value model by regressing V-lambda (Equation 8). The world model is fixed while behaviours are learned.
Example
// Create a Dreamer agent for an 8-dimensional observation and 2 continuous actions
var options = new DreamerOptions<double> { ObservationSize = 8, ActionSize = 2 };
var agent = new DreamerAgent<double>(options);
// Act: the agent filters its latent belief over the episode and returns an action in [-1, 1]
var observation = new Vector<double>(new double[8]);
var action = agent.SelectAction(observation);
Constructors
| Constructor | Summary |
|---|---|
DreamerAgent() |
Initializes a new instance with default settings. |
DreamerAgent(DreamerOptions<T> options) |
Initializes a Dreamer agent from options. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
T Train() |
|
Task TrainAsync() |
|
Task<Vector<T>> PredictAsync(Vector<T> input) |
|
Vector<T> Predict(Vector<T> input) |
The deterministic action for input taken as the first observation of an episode. |
Vector<T> SelectAction(Vector<T> observation, bool training) |
Advances the agent’s latent belief with observation and returns an action. |
byte[] Serialize() |
Auto-generated common model serialization surface. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void ResetEpisode() |
|
void StoreExperience(Vector<T> observation, Vector<T> action, T reward, Vector<T> nextObservation, bool done) |