WorldModelsAgent<T>
Models & Types · AiDotNet.ReinforcementLearning.Agents.WorldModels
World Models agent (Ha & Schmidhuber 2018): a VAE (V), an MDN-RNN (M) and a CMA-ES-evolved controller (C) trained inside M’s dream.
Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → WorldModelsAgent<T>
Implements: IGeneratedParameterRegistrar<T>
For Beginners
World Models is inspired by how humans learn: we build mental models of the world, then make decisions based on those models rather than raw sensory input.
Three components (V-M-C):
- V (VAE): Compresses observations into compact codes
- M (MDN-RNN): Learns what code, reward and episode end come next
- C (Controller): A tiny policy acting on the code and M’s memory
- Learning in Dreams: C is improved entirely inside M’s imagined rollouts
How It Works
V encodes an observation x into q(z|x) = N(mu, diag(exp(logvar))) and decodes z back, trained on ||x - D(z)||^2 + VAEBeta * KL(q(z|x) || N(0, I)) (appendix, “Variational Autoencoder”). M is an LSTM stack whose output parameterises, for every latent dimension, a mixture of NumMixtures Gaussians over z_{t+1}, plus the reward and the episode-end flag: P(z_{t+1}, r_{t+1}, d_{t+1} | a_t, z_t, h_t) (sections “MDN-RNN (M) Model” and “Iterative Training Procedure”). C is a_t = tanh(W_c [z_t h_t] + b_c), optionally with hidden tanh layers (section “Controller (C) Model”), and is trained by CMA-ES on the average cumulative reward of dream rollouts (appendix, “Evolution Strategies”; section “Training Inside of the Dream”).
What one Train call does. The paper trains V, then M on V’s encodings, then C, and repeats
the cycle as new experience arrives (“Iterative Training Procedure”). Each call advances that cycle by one unit
of work, so a call’s cost stays bounded:
- Vision: one optimizer step of V on the next BatchSize stored transitions of the current
epoch. After VAEEpochs full passes over the observations stored when the stage began, the stage moves to Memory.
- Memory: one optimizer step of M (BPTT through the LSTM) on the next BatchSize sequence
windows of the current epoch, each at most RNNSequenceLength long and never crossing an episode end; inputs are z sampled afresh from V’s posterior on every step, as the paper does to avoid overfitting a single sample. After RNNEpochs passes, the stage moves to Controller.
- Controller: one CMA-ES generation of ControllerPopulationSize candidates, each scored
by the mean cumulative predicted reward of ControllerEvaluationRollouts dream rollouts of up to RolloutLength steps, started from V’s encodings of stored observations; the controller becomes the search distribution’s mean. No external environment is needed. The search state persists across calls (and through Clone and serialization); after ControllerGenerations generations the cycle restarts at Vision.
The returned value is the loss of the unit just run: the VAE loss, the MDN-RNN loss, or the negated mean dream return of the generation’s candidates.
Example
// Create a World Models agent that learns in its own dreams
var options = new WorldModelsOptions<double>
{
ObservationWidth = 8, ObservationHeight = 1, ObservationChannels = 1,
ActionSize = 2, LatentSize = 4, RNNHiddenSize = 16
};
var agent = new WorldModelsAgent<double>(options);
var observation = new Vector<double>(new double[8]);
var action = agent.SelectAction(observation);
Constructors
| Constructor | Summary |
|---|---|
WorldModelsAgent() |
Initializes a new instance with default settings. |
WorldModelsAgent(WorldModelsOptions<T> options) |
Initializes a new World Models agent. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetMetrics() |
|
ModelMetadata<T> GetModelMetadata() |
|
T Train() |
|
Task TrainAsync() |
Asynchronously runs one Train call. |
Task<Vector<T>> PredictAsync(Vector<T> input) |
Asynchronously predicts the deterministic action for an observation. |
Vector<T> Predict(Vector<T> input) |
|
Vector<T> SelectAction(Vector<T> observation, bool training) |
|
byte[] Serialize() |
Auto-generated common model serialization surface. |
void Deserialize(byte[] data) |
Auto-generated common model deserialization surface. |
void LoadModel(string filepath) |
|
void ResetEpisode() |
|
void SaveModel(string filepath) |
|
void StoreExperience(Vector<T> observation, Vector<T> action, T reward, Vector<T> nextObservation, bool done) |