HawkLanguageModel<T>

Models & Types · AiDotNet.NeuralNetworks

Implements a Hawk language model: embedding + N pure RGLR blocks + layer norm + LM head.

Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → HawkLanguageModel<T>

For Beginners

Hawk is a pure-recurrent model that uses no attention at all, giving strict O(n) complexity and O(1) memory per token during generation.

How It Works

Hawk is the pure-recurrent variant from Google DeepMind (companion to Griffin), using only Real-Gated Linear Recurrence blocks without any attention. This gives strict O(n) complexity and O(1) memory per token during generation.

Reference: De et al., “Griffin: Mixing Gated Linear Recurrences with Local Attention”, 2024.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.TextGeneration,
    inputSize: 2048,
    outputSize: 256000);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new HawkLanguageModel<float>(architecture))
    .Build(trainX, trainY);
var logits = result.Predict(tokens);

Constructors

Constructor Summary
HawkLanguageModel(NeuralNetworkArchitecture<T> architecture, HawkOptions options, ILossFunction<T> lossFunction, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)

Properties

Property Summary
bool SupportsTraining { get; }
int ModelDimension { get; } Gets the model dimension (d_model).
int NumLayers { get; } Gets the number of Hawk blocks.
int VocabSize { get; } Gets the vocabulary size.

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()