DuelingCombinationLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Dueling DQN combination head (Wang et al. 2016, “Dueling Network Architectures for Deep Reinforcement Learning”). Splits the shared feature trunk’s output into a scalar state-value V(s) and an action-advantage vector A(s, a), then combines them as Q(s, a) = V(s) + (A(s, a) − mean_a A(s, a)) to enforce the identifiability constraint Eq. 9 of the paper.

Inheritance: LayerBase<T> → DuelingCombinationLayer<T>

Implements: IShapeContract

For Beginners

“Dueling” means the network learns the value of being in a state separately from the relative benefit of each action. V(s) answers “is this state good?”, A(s, a) answers “given this state, is action a better or worse than average?”. Combining them gives more stable training than predicting Q(s, a) directly with one big head — the value head doesn’t have to redundantly encode “this state is good” for every action.

How It Works

What this layer does: Reads [batch, featureDim] shared trunk features and emits [batch, actionSize] Q-values. Internally holds two parallel linear projections — a single-unit value head and an actionSize-unit advantage head — and combines them using the mean-subtraction form (paper §3, Eq. 9). The mean form is what every production Rainbow / Double-DQN implementation uses (e.g. Stable-Baselines3 DuelingQ, RLlib DuelingMixin) because the max-subtraction alternative is unstable under bootstrapping.

All forward arithmetic runs through Engine ops on the trainable tensor instances, so the gradient tape captures gradients for both heads automatically.

Constructors

Constructor Summary
DuelingCombinationLayer(int featureDim, int actionSize, int numAtoms, int[] valueHiddenWidths, int[] advantageHiddenWidths, int? seed) Initializes a dueling head: by default one Q-value per action with linear value and advantage heads, and optionally hidden stream layers and, for a distributional agent, one value per atom.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).