DecisionTransformerOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for Decision Transformer agents.

Inheritance: ModelOptions → ReinforcementLearningOptions<T> → DecisionTransformerOptions<T>

For Beginners

Decision Transformer is a radically different approach to RL. Instead of learning “what action is best”, it learns “what action was taken when the outcome was X”. Then at test time, you tell it “I want outcome X” and it generates actions.

Key innovation:

  • Sequence Modeling: Uses transformers (like GPT) instead of RL algorithms
  • Return Conditioning: Specify desired return, get action sequence
  • Offline-Friendly: Works excellently with fixed datasets
  • No Value Functions: No Q-networks or critics needed

Think of it like: “Show me examples of successful chess games, and I’ll learn to play moves that lead to success.”

Famous for: Berkeley/Meta research showing transformers can replace RL algorithms

How It Works

Decision Transformer treats RL as sequence modeling, using transformer architecture to model trajectories conditioned on desired returns.

Constructors

Constructor Summary
DecisionTransformerOptions() Creates the paper’s Gym training defaults.

Properties

Property Summary
Func<IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>>> OptimizerFactory { get; set; } Creates the transformer optimizer.
IOptimizer<T, Vector<T>, Vector<T>> Optimizer { get; set; } The optimizer used for updating network parameters.
bool IsContinuous { get; set; } Whether actions are continuous (tanh outputs, mean-squared error) or discrete (logits over one-hot actions, cross-entropy).
bool NormalizeStates { get; set; } Standardizes states using the retained offline dataset’s mean and population standard deviation.
double DropoutRate { get; set; } Dropout probability applied during training.
double ReturnScale { get; set; } Returns-to-go are divided by this before embedding.
double WeightDecay { get; set; } Decoupled AdamW weight decay when no optimizer factory is supplied.
double? TargetReturn { get; set; } The initial conditioning return; null uses the best completed episode return in the dataset.
int ActionSize { get; set; } Number of discrete actions, or the dimension of the action vector for a continuous action space.
int BufferSize { get; set; } Maximum number of transitions kept in the trajectory buffer.
int ContextLength { get; set; } Number of past time steps the model reads as context.
int EmbeddingDim { get; set; } Width of the token embeddings for returns-to-go, states and actions.
int LearningRateWarmupSteps { get; set; } Linear learning rate warmup steps.
int MaxEpisodeLength { get; set; } Size of the learned timestep embedding: the longest episode.
int NumHeads { get; set; } Number of attention heads.
int NumLayers { get; set; } Number of transformer blocks.
int StateSize { get; set; } Number of features in one observation of the environment state.