CQLOptions<T>

Options & Configuration · AiDotNet.Models.Options

Configuration options for Conservative Q-Learning (CQL) agent.

Inheritance: ModelOptions → CQLOptions<T>

For Beginners

CQL is designed for learning from logged data without trying new actions. This is useful when you have historical data but can’t experiment in the real environment (e.g., medical treatment, autonomous driving).

Key innovation:

  • Conservative Q-Learning: Penalizes Q-values for unseen actions to prevent overoptimistic estimates
  • Offline Learning: No environment interaction during training

Think of it like learning to drive from dashcam footage - you can’t try new maneuvers, so you need to be conservative about what you haven’t seen.

Based on SAC architecture with conservative regularization.

How It Works

CQL is an offline RL algorithm that learns from fixed datasets without environment interaction. It addresses overestimation by adding a conservative penalty to Q-values.

Constructors

Constructor Summary
CQLOptions() Creates options with the default values.

Properties

Property Summary
ILossFunction<T> QLossFunction { get; set; } Loss function used to train the Q-network.
List<int> PolicyHiddenLayers { get; set; } Hidden layer sizes of the policy network.
List<int> QHiddenLayers { get; set; } Hidden layer sizes of the Q-network.
T AlphaLearningRate { get; set; } Learning rate used when tuning the entropy temperature.
T CQLAlpha { get; set; } Weight of the conservative penalty that pushes down Q-values of actions outside the dataset.
T CQLTargetActionGap { get; set; } Target gap between out-of-distribution and dataset Q-values; with CQLLagrange the conservative weight is tuned to hold this gap.
T DiscountFactor { get; set; } Discount factor (gamma) applied to future rewards.
T InitialTemperature { get; set; } Initial entropy temperature (alpha) of the soft actor-critic objective.
T PolicyLearningRate { get; set; } Learning rate of the policy network.
T QLearningRate { get; set; } Learning rate of the Q-networks; with CQLLagrange it is also the Adam step size of the conservative multiplier (log alpha’) that dual ascent tunes toward CQLTargetActionGap.
T TargetUpdateTau { get; set; } Soft-update coefficient for the target networks: each update moves them this fraction of the way toward the online networks.
bool AutoTuneTemperature { get; set; } Whether to tune the entropy temperature automatically toward TargetEntropy.
bool CQLLagrange { get; set; } Whether to tune the conservative-penalty weight automatically (the Lagrangian form of CQL) instead of keeping CQLAlpha fixed.
double? TargetEntropy { get; set; } Target entropy for automatic temperature tuning (used when AutoTuneTemperature is on).
int ActionSize { get; set; } Number of discrete actions, or the dimension of the action vector for a continuous action space.
int BatchSize { get; set; } Number of samples in each training batch.
int BufferSize { get; set; } Maximum number of transitions kept in the offline replay buffer.
int CQLNumActions { get; set; } Number of actions sampled per state to estimate the conservative penalty’s log-sum-exp over Q-values.
int GradientSteps { get; set; } Number of gradient updates performed per training call.
int StateSize { get; set; } Number of features in one observation of the environment state.