DreamerOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for Dreamer agents (Hafner et al. 2020, “Dream to Control: Learning Behaviors by Latent Imagination”, ICLR 2020).

Inheritance: ModelOptions → ReinforcementLearningOptions<T> → DreamerOptions<T>

For Beginners

Dreamer learns a “mental model” of how the environment works, then uses that model to imagine future scenarios and learn from them - like practising a game in your head.

Key components:

  • World Model (RSSM): Learns environment dynamics in a compact latent space
  • Representation Model: Combines the memory with the current observation
  • Transition Model: Predicts the next latent state without seeing the observation
  • Reward / Discount Models: Predict rewards and whether the episode continues
  • Actor and Value Models: Learn the policy from imagined trajectories

Advantages: very sample efficient, because most learning happens in imagination.

How It Works

Dreamer learns a recurrent state-space model (RSSM) of the environment from sequences of experience and then learns an actor and a value model purely from trajectories imagined inside that model. The model state is the pair of a deterministic recurrent state h_t and a stochastic Gaussian state z_t; the representation model q(z_t | h_t, o_t) infers z_t from the observation, the transition model p(z_t | h_t) predicts it without the observation, and the observation, reward and discount models read the combined state (paper Section 4, Equations 9-10; Appendix A). The paper writes the representation model as p and the transition model as q; these docs use the RSSM’s posterior (q) and prior (p) names instead.

Constructors

Constructor Summary
DreamerOptions() Initializes the options with the hyperparameters of Hafner et al.

Properties

Property Summary
Func<IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>>> OptimizerFactory { get; set; } Gets or sets a factory for the optimizer each of this agent’s networks trains with.
T ActorLearningRate { get; set; } Gets the Adam learning rate of the action model.
T ValueLearningRate { get; set; } Gets the Adam learning rate of the value model.
double ContinueScale { get; set; } Gets the weight of the discount (episode-continuation) predictor’s log-likelihood in the world-model objective; zero disables the predictor.
double FreeNats { get; set; } Gets the free-nats threshold below which the KL regularizer is not optimized.
double KLScale { get; set; } Gets the scale beta of the KL regularizer between the representation (posterior) and transition (prior) models.
double Lambda { get; set; } Gets the lambda of the V-lambda value targets.
double RewardScale { get; set; } Gets the weight of the reward log-likelihood ln q(r_t | s_t) in the world-model objective.
int ActionSize { get; set; } Gets the width of the continuous action vector the agent emits.
int BatchLength { get; set; } Gets the length L of each sequence sampled from the replay data to train the world model.
int DeterministicSize { get; set; } Gets the size of the RSSM’s deterministic recurrent state h_t.
int HiddenSize { get; set; } Gets the width of the dense layers of the encoder, observation, reward, discount, action and value models.
int ImaginationHorizon { get; set; } Gets the imagination horizon H: how many steps the actor and value models look ahead inside the learned model.
int ObservationSize { get; set; } Gets the width of each observation vector the agent receives.
int StochasticSize { get; set; } Gets the size of the RSSM’s stochastic state z_t.