Trajectory<T>
Models & Types Ā· AiDotNet.ReinforcementLearning.Common
Represents a trajectory of experience for on-policy RL algorithms (PPO, A2C, etc.).
For Beginners
A trajectory is like recording a game session. It contains:
- Every state you saw
- Every action you took
- Every reward you got
- Additional info (value estimates, action probabilities)
On-policy algorithms like PPO collect these trajectories, learn from them immediately, then throw them away and collect new ones. This is different from DQN which stores experiences in a replay buffer and samples from them multiple times.
How It Works
A trajectory is a sequence of states, actions, and rewards collected by an agent interacting with an environment. Unlike experience replay (used in DQN), trajectories are used immediately for training and then discarded in on-policy algorithms.
Constructors
| Constructor | Summary |
|---|---|
Trajectory() |
Initializes an empty trajectory. |
Properties
| Property | Summary |
|---|---|
List<T> Advantages { get; set; } |
Computed advantages (used during training). |
List<T> BootstrapValues { get; set; } |
V(s_{t+1}) for each truncated step (zero for the other steps). |
List<T> LogProbs { get; set; } |
Log probabilities of actions taken (for policy gradient). |
List<T> Returns { get; set; } |
Computed returns (discounted sum of rewards). |
List<T> Rewards { get; set; } |
Rewards received during the trajectory. |
List<T> Values { get; set; } |
Value estimates for each state (from critic). |
List<Vector<T>> Actions { get; set; } |
Actions taken during the trajectory. |
List<Vector<T>> States { get; set; } |
States observed during the trajectory. |
List<bool> Dones { get; set; } |
Whether each step was terminal: the episode reached a terminal state, so nothing is bootstrapped after it. |
List<bool> Truncations { get; set; } |
Whether each step was truncated: the episode was cut off (a time limit or the end of the rollout) in a non-terminal state, so the return after it is bootstrapped from BootstrapValues. |
int Length { get; } |
Gets the number of steps in the trajectory. |
Methods
| Method | Summary |
|---|---|
Vector<T> NextValues() |
Vāt per step, the value bootstrapped after step t: zero after a terminal step, the stored V(s{t+1}) after a truncated one, and V of the next stored step otherwise. |
bool[] ChainCuts() |
Whether the return chain stops after each step: a terminal or truncated step, or the last one. |
void AddStep(Vector<T> state, Vector<T> action, T reward, T value, T logProb, bool done) |
Adds a step to the trajectory. |
void AddStep(Vector<T> state, Vector<T> action, T reward, T value, T logProb, bool terminated, bool truncated, T bootstrapValue) |
Adds a step that ended its episode by termination, by truncation, or neither. |
void Clear() |
Clears the trajectory. |
void TruncateLastStep(T bootstrapValue) |
Marks the last step as truncated with the given V(s_{t+1}) when it ended neither by termination nor by truncation: the rollout itself cut the episode off there. |