EveryVisitMonteCarloAgent<T>

Models & Types ยท AiDotNet.ReinforcementLearning.Agents.MonteCarlo

Every-Visit Monte Carlo agent that updates all visits to states in an episode.

Inheritance: ReinforcementLearningAgentBase<T> โ†’ EveryVisitMonteCarloAgent<T>

Implements: IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

Every-Visit Monte Carlo learns by playing complete episodes (from start to finish) and then averaging the total reward received. Unlike First-Visit MC which only counts the first time a state is seen, this counts every visit. Think of it like a student who reviews every practice problem, not just the first attempt. This gives more data points per episode but with potentially correlated samples. Good for episodic tasks like board games where you learn from complete games.

Example

// Create an Every-Visit Monte Carlo agent for episodic tasks
var options = new MonteCarloOptions<double> { StateSize = 4, ActionSize = 2 };
var agent = new EveryVisitMonteCarloAgent<double>(options);
            
// Select an action using epsilon-greedy over learned Q-values
var state = new Vector<double>(new double[] { 0.5, -0.3, 1.0, 0.2 });
var action = agent.SelectAction(state);

Constructors

Constructor Summary
EveryVisitMonteCarloAgent() Initializes a new instance with default settings.
EveryVisitMonteCarloAgent(MonteCarloOptions<T> options)

Properties

Property Summary
int FeatureCount { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
T Train()
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction)
Vector<T> SelectAction(Vector<T> state, bool training)
byte[] Serialize() Auto-generated common model serialization surface.
void ApplyGradients(Vector<T> gradients, T learningRate)
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void ResetEpisode()
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done)