UpperConfidenceBoundExploration<T>

Models & Types · AiDotNet.ReinforcementLearning.Policies.Exploration

Upper Confidence Bound (UCB) exploration for discrete action spaces. Balances exploration and exploitation using confidence intervals: UCB(a) = Q(a) + c * √(ln(t) / N(a))

Inheritance: ExplorationStrategyBase<T> → UpperConfidenceBoundExploration<T>

Example

var random = new Random(42);
var state = new Vector<double>(new double[] { 0.1, -0.4, 0.7, 0.2 });
// Create UCB exploration that favors under-explored actions
var exploration = new UpperConfidenceBoundExploration<double>(explorationConstant: 2.0);
            
// Select action using upper confidence bound: Q(a) + c * sqrt(ln(t) / N(a))
var policyAction = new Vector<double>(new double[] { 1.5, 0.8, 2.3 });
var action = exploration.GetExplorationAction(state, policyAction, actionSpaceSize: 3, random);

Constructors

Constructor Summary
UpperConfidenceBoundExploration(double explorationConstant) Initializes a new instance of the Upper Confidence Bound exploration strategy.

Properties

Property Summary
double ExplorationConstant { get; } Gets the current exploration constant.
int TotalSteps { get; } Gets the total number of steps taken.

Methods

Method Summary
Vector<T> GetExplorationAction(Vector<T> state, Vector<T> policyAction, int actionSpaceSize, Random random) Selects action using UCB: action with highest Q(a) + c * √(ln(t) / N(a))
void Reset() Resets action counts and total steps.
void Update() Updates internal parameters (UCB is count-based, no explicit decay).