QMIXAgent<T>

Models & Types · AiDotNet.ReinforcementLearning.Agents.QMIX

QMIX (Rashid et al. 2018): cooperative multi-agent Q-learning whose joint action-value is a monotonic mix of the agents’ own values.

Inheritance: ReinforcementLearningAgentBase<T> → DeepReinforcementLearningAgentBase<T> → QMIXAgent<T>

Implements: IGeneratedParameterRegistrar<T>, IGradientComputable<T, Vector<T>, Vector<T>>

For Beginners

Each player estimates how good each of its moves is from what it can see; a coach combines the estimates of the moves actually played into a team score, in a way where a better move for any player can only raise the team’s score. So each player picking its own best move also picks the team’s best plan.

How It Works

Every agent runs one shared recurrent network (a ReLU layer, a GRU cell and a linear layer, fed its observation, its last action and its id) and acts ε-greedily on its own values. In training, the values of the actions taken are mixed into Q_tot by a network whose weights hypernetworks generate from the global state and keep non-negative, so Q_tot rises with every agent’s value; the agent and mixing networks train together on the TD error of Q_tot against target networks, over batches of whole episodes unrolled from the episode start.

It follows the paper, with the authors’ code (pymarl) where the paper is silent; pymarl’s own configuration is available through HypernetLayers, DoubleQ and MeanLoss. QmixReferenceTests matches one training step of pymarl’s QMixer and RNNAgent to 1e-9 in both configurations.

Usage: each step, call SelectActionForAgent for every agent, step the environment, and store the joint transition with StoreMultiAgentExperience; call Train after each step. A transition that ends the episode (done, or truncated at a time limit) completes it, adds it to the buffer and resets the agents’ recurrent states; Train runs one update per completed episode once the buffer holds a batch.

Example

var agent = new QMIXAgent<double>(new QMIXOptions<double> { NumAgents = 3, StateSize = 8, ActionSize = 4, GlobalStateSize = 12 });
var action = agent.SelectActionForAgent(0, observation);

Constructors

Constructor Summary
QMIXAgent() Initializes a new instance with default settings.
QMIXAgent(QMIXOptions<T> options) Creates the native recurrent agent with the configured joint optimizer.
QMIXAgent(QMIXOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a QMIX agent.

Properties

Property Summary
double Epsilon { get; } The current exploration rate: linear from EpsilonStart to EpsilonEnd over EpsilonAnnealSteps, as of the episode’s start (pymarl).
int FeatureCount { get; }

Methods

Method Summary
Dictionary<string, T> GetMetrics()
ModelMetadata<T> GetModelMetadata()
T Train() Runs one QMIX update when an episode has completed since the last one and the buffer holds a batch of episodes; otherwise returns zero.
Task TrainAsync() Trains asynchronously (see Train).
Task<Vector<T>> PredictAsync(Vector<T> input) Predicts asynchronously (see Predict).
Vector<T> ComputeGradients(Vector<T> input, Vector<T> target, ILossFunction<T> lossFunction) Computes the mixing network’s parameter gradients of a loss on one input, without updating them.
Vector<T> Predict(Vector<T> input) Agent 0’s greedy action for an observation at the start of an episode (a zero recurrent state and no last action), without changing any agent’s recurrent state.
Vector<T> SelectAction(Vector<T> state, bool training) Selects agent 0’s action (QMIX’s agents act through SelectActionForAgent).
Vector<T> SelectActionForAgent(int agentId, Vector<T> state, bool training) Selects one agent’s action, ε-greedily when training, advancing that agent’s recurrent state.
byte[] Serialize() Auto-generated common model serialization surface.
void Deserialize(byte[] data) Auto-generated common model deserialization surface.
void LoadModel(string filepath)
void ResetEpisode() Starts a new episode: every agent’s recurrent state and last action reset, and any partial episode dropped.
void SaveModel(string filepath)
void StoreExperience(Vector<T> state, Vector<T> action, T reward, Vector<T> nextState, bool done) QMIX learns from joint transitions: use StoreMultiAgentExperience.
void StoreMultiAgentExperience(List<Vector<T>> agentStates, List<Vector<T>> agentActions, T teamReward, List<Vector<T>> nextAgentStates, Vector<T> globalState, Vector<T> nextGlobalState, bool done, bool truncated) Stores one joint transition of the episode being collected.