ReinforcementLearningAgentBase<T>
Base Classes · AiDotNet.ReinforcementLearning.Agents
Base class for all reinforcement learning agents, providing common functionality and structure.
Implements: IModelShape, IParameterChunkSource<T>, IParameterManifestProvider, IRLAgent<T>
Known derived types: DeepReinforcementLearningAgentBase<T>, DoubleQLearningAgent<T>, DynaQAgent<T>, DynaQPlusAgent<T>, EpsilonGreedyBanditAgent<T>, EveryVisitMonteCarloAgent<T>, ExpectedSARSAAgent<T>, FirstVisitMonteCarloAgent<T>, GradientBanditAgent<T>, LSPIAgent<T>, LSTDAgent<T>, LinearQLearningAgent<T>, LinearSARSAAgent<T>, ModifiedPolicyIterationAgent<T>, MonteCarloExploringStartsAgent<T>, NStepQLearningAgent<T>, NStepSARSAAgent<T>, OffPolicyMonteCarloAgent<T>, OnPolicyMonteCarloAgent<T>, PolicyIterationAgent<T>, PrioritizedSweepingAgent<T>, QLambdaAgent<T>, SARSAAgent<T>, SARSALambdaAgent<T>, TabularActorCriticAgent<T>, TabularQLearningAgent<T>, ThompsonSamplingAgent<T>, TradingAgentBase<T>, UCBBanditAgent<T>, ValueIterationAgent<T>, WatkinsQLambdaAgent<T>
For Beginners
This is the foundation for all RL agents in AiDotNet.
Think of this base class as the blueprint that defines what every RL agent must be able to do:
- Select actions based on observations
- Store experiences for learning
- Train/update from experiences
- Save and load trained models
- Integrate with AiDotNet’s neural networks and optimizers
All specific RL algorithms (DQN, PPO, SAC, etc.) inherit from this base and implement their own unique learning logic while sharing common functionality.
How It Works
This abstract base class defines the core structure that all RL agents must follow, ensuring consistency across different RL algorithms while allowing for specialized implementations. It integrates deeply with AiDotNet’s existing architecture, using Vector, Matrix, and Tensor types, and following established patterns like OptimizerBase and NeuralNetworkBase.
Properties
| Property | Summary |
|---|---|
ILossFunction<T> DefaultLossFunction { get; } |
Gets the default loss function for this agent. |
ParameterLayoutSnapshot ParameterLayout { get; } |
|
bool SupportsParameterInitialization { get; } |
|
int ActionDimension { get; } |
Gets the number of action dimensions. |
int FeatureCount { get; } |
Gets the number of input features (state dimensions). |
long ParameterCount { get; } |
Gets the number of parameters in the agent. |
string[] FeatureNames { get; } |
Gets the names of input features. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, T> GetFeatureImportance() |
Gets feature importance scores. |
Dictionary<string, T> GetMetrics() |
Gets the current training metrics. |
DynamicShapeInfo GetDynamicShapeInfo() |
|
IEnumerable<ParameterChunk<T>> GetParameterStateChunks() |
Yields this agent’s registered state as per-component chunks, each carrying the stable ID and role of the component it came from. |
IEnumerable<Tensor<T>> GetParameterChunks() |
Streams state in manifest order without flattening tensor-backed components. |
IEnumerable<int> GetActiveFeatureIndices() |
Gets the indices of active features. |
IFullModel<T, Vector<T>, Vector<T>> Clone() |
Clones the agent. |
IFullModel<T, Vector<T>, Vector<T>> DeepCopy() |
Creates a deep copy of the agent. |
IFullModel<T, Vector<T>, Vector<T>> WithParameters(Vector<T> parameters) |
Creates a new instance with the specified parameters. |
ModelMetadata<T> GetModelMetadata() |
Gets model metadata. |
ModelOptions GetOptions() |
|
T Train() |
Performs one training step, updating the agent’s policy/value function. |
Vector<T> GetParameters() |
|
Vector<T> Predict(Vector<T> input) |
Makes a prediction using the trained agent. |
Vector<T> SanitizeParameters(Vector<T> parameters) |
|
Vector<T> SelectAction(Vector<T> state, bool training) |
Selects an action given the current state observation. |
bool IsFeatureUsed(int featureIndex) |
Checks if a feature is used by the agent. |
byte[] Serialize() |
Serializes the agent to bytes. |
int[] GetInputShape() |
|
int[] GetOutputShape() |
|
void Deserialize(byte[] data) |
Deserializes the agent from bytes. |
void Dispose() |
Disposes of resources used by the agent. |
void LoadModel(string filepath) |
Loads the agent’s state from a file, stripping the AIMF header if present. |
void LoadState(Stream stream) |
Loads the agent’s state (parameters and configuration) from a stream. |
void ResetEpisode() |
Resets episode-specific state (if any). |
void SaveModel(string filepath) |
Saves the agent’s state to a file with an AIMF envelope header. |
void SaveState(Stream stream) |
Saves the agent’s current state (parameters and configuration) to a stream. |
void SetActiveFeatureIndices(IEnumerable<int> indices) |
Sets the active feature indices. |
void SetParameterChunks(IEnumerable<Tensor<T>> chunks) |
Restores canonical tensor chunks on their device, preserving custom setters and hooks. |
void SetParameters(Vector<T> parameters) |
Sets the agent’s parameters. |
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) |
Stores an experience tuple for later learning. |
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done, bool[] nextLegalActions) |
Stores a transition, telling termination apart from truncation. |
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool terminated, bool truncated) |
Stores a transition, telling termination apart from truncation. |
void Train(Vector<T> state, Vector<T> target) |
Trains the agent on a single (state, target) supervised pair by translating it into a one-step RL transition and dispatching through the standard StoreExperience + Train pipeline. |