WorldModelsAgent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.WorldModels

World Models agent (Ha & Schmidhuber 2018): a VAE (V), an MDN-RNN (M) and a CMA-ES-evolved controller (C) trained inside M’s dream.

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → WorldModelsAgent<T>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

World Models is inspired by how humans learn: we build mental models of the world, then make decisions based on those models rather than raw sensory input.

Three components (V-M-C):

  • V (VAE): Compresses observations into compact codes
  • M (MDN-RNN): Learns what code, reward and episode end come next
  • C (Controller): A tiny policy acting on the code and M’s memory
  • Learning in Dreams: C is improved entirely inside M’s imagined rollouts

How It Works

V encodes an observation x into q(z|x) = N(mu, diag(exp(logvar))) and decodes z back, trained on ||x - D(z)||^2 + VAEBeta * KL(q(z|x) || N(0, I)) (appendix, “Variational Autoencoder”). M is an LSTM stack whose output parameterises, for every latent dimension, a mixture of NumMixtures Gaussians over z_{t+1}, plus the reward and the episode-end flag: P(z_{t+1}, r_{t+1}, d_{t+1} | a_t, z_t, h_t) (sections “MDN-RNN (M) Model” and “Iterative Training Procedure”). C is a_t = tanh(W_c [z_t h_t] + b_c), optionally with hidden tanh layers (section “Controller (C) Model”), and is trained by CMA-ES on the average cumulative reward of dream rollouts (appendix, “Evolution Strategies”; section “Training Inside of the Dream”).

What one Train call does. The paper trains V, then M on V’s encodings, then C, and repeats the cycle as new experience arrives (“Iterative Training Procedure”). Each call advances that cycle by one unit of work, so a call’s cost stays bounded:

  • Vision: one optimizer step of V on the next BatchSize stored transitions of the current

epoch. After VAEEpochs full passes over the observations stored when the stage began, the stage moves to Memory.

  • Memory: one optimizer step of M (BPTT through the LSTM) on the next BatchSize sequence

windows of the current epoch, each at most RNNSequenceLength long and never crossing an episode end; inputs are z sampled afresh from V’s posterior on every step, as the paper does to avoid overfitting a single sample. After RNNEpochs passes, the stage moves to Controller.

  • Controller: one CMA-ES generation of ControllerPopulationSize candidates, each scored

by the mean cumulative predicted reward of ControllerEvaluationRollouts dream rollouts of up to RolloutLength steps, started from V’s encodings of stored observations; the controller becomes the search distribution’s mean. No external environment is needed. The search state persists across calls (and through Clone and serialization); after ControllerGenerations generations the cycle restarts at Vision.

The returned value is the loss of the unit just run: the VAE loss, the MDN-RNN loss, or the negated mean dream return of the generation’s candidates.

Example

// Create a World Models agent that learns in its own dreams
var options = new WorldModelsOptions<double>
{
    ObservationWidth = 8, ObservationHeight = 1, ObservationChannels = 1,
    ActionSize = 2, LatentSize = 4, RNNHiddenSize = 16
};
var agent = new WorldModelsAgent<double>(options);
            
var observation = new Vector<double>(new double[8]);
var action = agent.SelectAction(observation);

Constructors

Constructor Summary
WorldModelsAgent() Initializes a new instance with default settings.
WorldModelsAgent(WorldModelsOptions<T> options) Initializes a new World Models agent.

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train()
Task TrainAsync() Asynchronously runs one Train call.
Task<Vector<T>> PredictAsync(Vector<T> input) Asynchronously predicts the deterministic action for an observation.
Vector<T> Predict(Vector<T> input)
Vector<T> SelectAction(Vector<T> observation, bool training)
byte[] Serialize() Auto-generated common model serialization surface.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void ResetEpisode()
void SaveModel(string filepath)
void StoreExperience(Vector<T> observation, Vector<T> action, T reward, Vector<T> nextObservation, bool done)