DecisionTransformerAgent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.DecisionTransformer

Decision Transformer agent for offline reinforcement learning.

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → DecisionTransformerAgent<T>

Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

Instead of learning “what’s the best action”, Decision Transformer learns “what action was taken when the outcome was X”. At test time, you specify the desired outcome, and it generates the action sequence.

Key innovation:

  • Return Conditioning: Specify target return, get actions that achieve it
  • Sequence Modeling: Uses transformers like GPT for temporal dependencies
  • No RL Updates: Just supervised learning on (return, state, action) sequences
  • Offline-First: Designed for learning from fixed datasets

Think of it as: “Show me examples of successful games, and I’ll learn to generate moves that lead to that level of success.”

Famous for: Berkeley/Meta research simplifying RL to sequence modeling

How It Works

Decision Transformer treats RL as sequence modeling, using transformer architecture to predict actions conditioned on desired returns-to-go.

The network (section 3) embeds the last K timesteps as 3K tokens, (return-to-go, state, action) per timestep, with a linear layer per modality plus a learned per-timestep embedding, normalizes them, and runs a causally masked GPT (3 layers, 1 head, 128 dimensions, dropout 0.1 by default, the Gym model of appendix A.2); the hidden state of each state token predicts that timestep’s action. Training regresses the dataset’s actions (mean-squared error, or cross-entropy for discrete actions) on K-step segments; acting conditions on a target return that each reward decrements.

Example

// Create a Decision Transformer for offline RL via sequence modeling
var options = new DecisionTransformerOptions<double> { StateSize = 4, ActionSize = 2, ContextLength = 20 };
var agent = new DecisionTransformerAgent<double>(options);
            
// Select an action conditioned on the desired return
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);

Constructors

Constructor Summary
DecisionTransformerAgent()
DecisionTransformerAgent(DecisionTransformerOptions<T> options, IOptimizer<T, Vector<T>, Vector<T>> optimizer)

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train() One gradient step (section 3, “Training”): a minibatch of K-step segments, each predicting its actions from the state tokens, with the mean-squared error (continuous) or cross-entropy (discrete) averaged over the timesteps.
Task TrainAsync()
Task<Vector<T>> PredictAsync(Vector<T> input)
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) Computes gradients of the loss with respect to this agent’s parameters, without updating them.
Vector<T> Predict(Vector<T> input)
Vector<T> SelectAction(Vector<T> state, bool training) Selects an action for the episode’s next state, conditioned on the return-to-go still to collect: the configured TargetReturn at the start of an episode, decremented by each reward passed to StoreExperience.
Vector<T> SelectActionWithReturn(Vector<T> state, T targetReturn, bool training) Select action conditioned on desired return-to-go (Algorithm 1’s evaluation loop): appends (R, s, t) to the context, keeps its last K timesteps, and returns the action predicted at the newest state token.
byte[] Serialize() Auto-generated common model serialization surface.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void LoadOfflineData(List<List<ValueTuple<Vector<T>, Vector<T>, T>>> trajectories) Loads offline trajectories into the dataset, computing each step’s return-to-go (the sum of the rewards from that step to the end) and timestep.
void ResetEpisode()
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) Collects the episode; when it ends it joins the offline dataset (with its returns-to-go).
void Train(Vector<T> state, Vector<T> target) Supervised (online) training entry from IFullModel: one step predicting target (the action) from a one-timestep sequence of state at return-to-go 0 and timestep 0.