IQLOptions<T>
Options & Configuration · AiDotNet.Models.Options
Configuration options for Implicit Q-Learning (IQL) agent.
Inheritance: ModelOptions → IQLOptions<T>
For Beginners
IQL is designed for offline learning (learning from fixed datasets). Unlike CQL which adds penalties, IQL uses a clever trick called “expectile regression” to avoid overestimation.
Key innovation:
- Expectile Regression: Focus on upper quantiles of value distribution
- Implicit Policy Extraction: No explicit max over actions
- Simpler than CQL: Fewer hyperparameters to tune
Think of it like learning the “typical good outcome” rather than the “best possible outcome” which helps avoid being too optimistic about unseen situations.
Advantages: Simpler, more stable than CQL in many cases
How It Works
IQL is an offline RL algorithm that avoids explicit policy constraints or conservative regularization. Instead, it uses expectile regression to extract a policy from the value function.
Constructors
| Constructor | Summary |
|---|---|
IQLOptions() |
Creates options with the default values. |
Properties
| Property | Summary |
|---|---|
ILossFunction<T> QLossFunction { get; set; } |
Loss function used to train the Q-network. |
List<int> PolicyHiddenLayers { get; set; } |
Hidden layer sizes of the policy network. |
List<int> QHiddenLayers { get; set; } |
Hidden layer sizes of the Q-network. |
List<int> ValueHiddenLayers { get; set; } |
Hidden layer sizes of the value network. |
T DiscountFactor { get; set; } |
Discount factor (gamma) applied to future rewards. |
T PolicyLearningRate { get; set; } |
Learning rate of the policy network. |
T QLearningRate { get; set; } |
Learning rate of the Q-network. |
T TargetUpdateTau { get; set; } |
Soft-update coefficient for the target networks: each update moves them this fraction of the way toward the online networks. |
T Temperature { get; set; } |
Inverse temperature of advantage-weighted regression: higher values follow high-advantage actions more closely. |
T ValueLearningRate { get; set; } |
Learning rate of the value network. |
double Expectile { get; set; } |
Expectile (tau) of the value regression; above 0.5 it approximates the maximum over in-dataset actions (0.7-0.9 is typical). |
int ActionSize { get; set; } |
Number of discrete actions, or the dimension of the action vector for a continuous action space. |
int BatchSize { get; set; } |
Number of samples in each training batch. |
int BufferSize { get; set; } |
Maximum number of transitions kept in the offline replay buffer. |
int StateSize { get; set; } |
Number of features in one observation of the environment state. |
Methods
| Method | Summary |
|---|---|
void Validate() |
Validates that required properties are set. |