DreamerOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for Dreamer agents (Hafner et al. 2020, “Dream to Control: Learning Behaviors by Latent Imagination”, ICLR 2020).
Inheritance: ModelOptions → ReinforcementLearningOptions<T> → DreamerOptions<T>
For Beginners
Dreamer learns a “mental model” of how the environment works, then uses that model to imagine future scenarios and learn from them - like practising a game in your head.
Key components:
- World Model (RSSM): Learns environment dynamics in a compact latent space
- Representation Model: Combines the memory with the current observation
- Transition Model: Predicts the next latent state without seeing the observation
- Reward / Discount Models: Predict rewards and whether the episode continues
- Actor and Value Models: Learn the policy from imagined trajectories
Advantages: very sample efficient, because most learning happens in imagination.
How It Works
Dreamer learns a recurrent state-space model (RSSM) of the environment from sequences of experience and then learns an actor and a value model purely from trajectories imagined inside that model. The model state is the pair of a deterministic recurrent state h_t and a stochastic Gaussian state z_t; the representation model q(z_t | h_t, o_t) infers z_t from the observation, the transition model p(z_t | h_t) predicts it without the observation, and the observation, reward and discount models read the combined state (paper Section 4, Equations 9-10; Appendix A). The paper writes the representation model as p and the transition model as q; these docs use the RSSM’s posterior (q) and prior (p) names instead.
Constructors
| Constructor | Summary |
|---|---|
DreamerOptions() |
Initializes the options with the hyperparameters of Hafner et al. |
Properties
| Property | Summary |
|---|---|
Func<IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>>> OptimizerFactory { get; set; } |
Gets or sets a factory for the optimizer each of this agent’s networks trains with. |
T ActorLearningRate { get; set; } |
Gets the Adam learning rate of the action model. |
T ValueLearningRate { get; set; } |
Gets the Adam learning rate of the value model. |
double ContinueScale { get; set; } |
Gets the weight of the discount (episode-continuation) predictor’s log-likelihood in the world-model objective; zero disables the predictor. |
double FreeNats { get; set; } |
Gets the free-nats threshold below which the KL regularizer is not optimized. |
double KLScale { get; set; } |
Gets the scale beta of the KL regularizer between the representation (posterior) and transition (prior) models. |
double Lambda { get; set; } |
Gets the lambda of the V-lambda value targets. |
double RewardScale { get; set; } |
Gets the weight of the reward log-likelihood ln q(r_t | s_t) in the world-model objective. |
int ActionSize { get; set; } |
Gets the width of the continuous action vector the agent emits. |
int BatchLength { get; set; } |
Gets the length L of each sequence sampled from the replay data to train the world model. |
int DeterministicSize { get; set; } |
Gets the size of the RSSM’s deterministic recurrent state h_t. |
int HiddenSize { get; set; } |
Gets the width of the dense layers of the encoder, observation, reward, discount, action and value models. |
int ImaginationHorizon { get; set; } |
Gets the imagination horizon H: how many steps the actor and value models look ahead inside the learned model. |
int ObservationSize { get; set; } |
Gets the width of each observation vector the agent receives. |
int StochasticSize { get; set; } |
Gets the size of the RSSM’s stochastic state z_t. |