FalconMambaLanguageModel<T>
Models & Types · AiDotNet.NeuralNetworks
Implements a Falcon Mamba language model: embedding + N MambaBlock blocks + RMS norm + LM head.
Inheritance: NeuralNetworkBase<T> → DeclaredModelLayoutBase<T> → TokenLanguageModelLayoutBase<T> → FalconMambaLanguageModel<T>
For Beginners
Falcon Mamba is a 7-billion parameter language model that works without attention mechanisms (unlike GPT or LLaMA). Instead, it uses the Mamba architecture which processes sequences with constant memory, meaning it can handle infinitely long conversations without slowing down. This makes it uniquely suited for applications requiring very long context windows.
How It Works
Falcon Mamba from TII (Technology Innovation Institute) is a pure Mamba-based language model trained at 7B scale, achieving competitive results with Transformer-based models while maintaining constant memory during generation regardless of sequence length.
Reference: Zuo et al., “Falcon Mamba: The First Competitive Attention-free 7B Language Model”, 2024.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 8, outputSize: 4);
var tokens = Tensor<float>.CreateRandom(new[] { 1, 128 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new FalconMambaLanguageModel<float>(architecture))
.Build(trainX, trainY);
var logits = result.Predict(tokens);
Constructors
| Constructor | Summary |
|---|---|
FalconMambaLanguageModel(NeuralNetworkArchitecture<T> architecture, FalconMambaOptions options, ILossFunction<T> lossFunction) |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int ModelDimension { get; } |
Gets the model dimension (d_model). |
int NumLayers { get; } |
Gets the number of Falcon Mamba blocks. |
int VocabSize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |