GatedDeltaNetLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a full Gated DeltaNet language model: token embedding + N GatedDeltaNetLayer blocks + RMS norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → GatedDeltaNetLanguageModel<T>
For Beginners
Gated DeltaNet improves on Mamba2 by using delta rules that can both add and remove information from memory, leading to better sequence modeling.
How It Works
Gated DeltaNet combines linear attention with gated delta rules for efficient sequence modeling. The delta rule update allows the model to both write new associations and erase old ones in its memory, unlike standard linear attention which can only accumulate.
Reference: Yang et al., “Gated Delta Networks: Improving Mamba2 with Delta Rule”, 2024.
Example
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new GatedDeltaNetLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
GatedDeltaNetLanguageModel(NeuralNetworkArchitecture<T> architecture, GatedDeltaNetOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Gated DeltaNet blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |