DreamerAgent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.Dreamer

Dreamer agent (Hafner et al. 2020, “Dream to Control: Learning Behaviors by Latent Imagination”).

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → DreamerAgent<T>

For Beginners

Dreamer learns a “mental model” of how the environment works, then uses that model to imagine future scenarios and plan actions - like chess players thinking multiple moves ahead.

Key components:

  • Representation Model: Combines the agent’s memory with the current observation
  • Transition Model: Predicts the next latent state without seeing the observation
  • Reward / Discount Models: Predict rewards and whether the episode continues
  • Value Model: Estimates how good an imagined state is
  • Action Model: Learns the policy entirely in imagination

Advantages: Sample efficient, works with images, enables planning

How It Works

World model (Section 4, Equations 9-10; Appendix A). A recurrent state-space model (RSSM, Hafner et al. 2018) whose model state s_t = (h_t, z_t) pairs a deterministic GRU state h_t (DeterministicSize) with a diagonal-Gaussian stochastic state z_t (StochasticSize): h_t = GRU(h_{t-1}, [z_{t-1}, a_{t-1}]); transition (prior) p(z_t | h_t); representation (posterior) q(z_t | h_t, e_t) with e_t = enc(o_t); observation model q(o_t | s_t); reward model q(r_t | s_t); and, for tasks with early termination, a discount model trained toward the soft labels 0 and gamma (Section 3, Appendix A). The model maximises Equation 10 on BatchLength-step sequences drawn from the replay data: reconstruction + RewardScale * reward

  • ContinueScale * discount log-likelihoods, minus KLScale * KL(posterior || prior) clipped below FreeNats. (The paper writes the representation model as p and the transition model as q; this class uses the RSSM’s posterior / prior names.)

Behaviour learning (Section 3, Equations 2-8). From every posterior state of the sampled sequences the agent imagines ImaginationHorizon steps with the transition model and a tanh-transformed Gaussian action model (Equation 3), computes the V-lambda targets of Equation 6, and trains the action model by backpropagating V-lambda through the imagined dynamics (Equation 7) and the value model by regressing V-lambda (Equation 8). The world model is fixed while behaviours are learned.

Example

// Create a Dreamer agent for an 8-dimensional observation and 2 continuous actions
var options = new DreamerOptions<double> { ObservationSize = 8, ActionSize = 2 };
var agent = new DreamerAgent<double>(options);
            
// Act: the agent filters its latent belief over the episode and returns an action in [-1, 1]
var observation = new Vector<double>(new double[8]);
var action = agent.SelectAction(observation);

Constructors

Constructor Summary
DreamerAgent() Initializes a new instance with default settings.
DreamerAgent(DreamerOptions<T> options) Initializes a Dreamer agent from options.

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train()
Task TrainAsync()
Task<Vector<T>> PredictAsync(Vector<T> input)
Vector<T> Predict(Vector<T> input) The deterministic action for input taken as the first observation of an episode.
Vector<T> SelectAction(Vector<T> observation, bool training) Advances the agent’s latent belief with observation and returns an action.
byte[] Serialize() Auto-generated common model serialization surface.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void ResetEpisode()
void StoreExperience(Vector<T> observation, Vector<T> action, T reward, Vector<T> nextObservation, bool done)