GatedDeltaNetLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a full Gated DeltaNet language model: token embedding + N GatedDeltaNetLayer blocks + RMS norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → GatedDeltaNetLanguageModel<T>

For Beginners

Gated DeltaNet improves on Mamba2 by using delta rules that can both add and remove information from memory, leading to better sequence modeling.

How It Works

Gated DeltaNet combines linear attention with gated delta rules for efficient sequence modeling. The delta rule update allows the model to both write new associations and erase old ones in its memory, unlike standard linear attention which can only accumulate.

Reference: Yang et al., “Gated Delta Networks: Improving Mamba2 with Delta Rule”, 2024.

Example

var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new GatedDeltaNetLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
GatedDeltaNetLanguageModel(NeuralNetworkArchitecture<T> architecture, GatedDeltaNetOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Gated DeltaNet blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()