LagLlama<T>

Models & Types · AiDotNet.Finance.Forecasting.Foundation

Lag-Llama foundation model for probabilistic time series forecasting.

Inheritance: NeuralNetworkBase<T> → FinancialModelBase<T> → ForecastingModelBase<T> → LagLlama<T>

For Beginners

Lag-Llama brings LLM innovations to time series forecasting:

The Lag Feature Innovation: Instead of just looking at recent values, Lag-Llama creates features from specific past points:

  • Lag-1: Yesterday’s value (immediate trend)
  • Lag-7: Same day last week (weekly pattern)
  • Lag-365: Same day last year (annual pattern)

This lets the model explicitly see patterns at different time scales without needing a very long context window.

Llama Architecture Adaptations: Lag-Llama adopts key Llama innovations:

  • RMSNorm: Simpler, faster layer normalization
  • SwiGLU: Improved MLP activation function
  • RoPE: Rotary Position Embeddings for better position encoding
  • Causal Attention: Each position only sees earlier positions

Probabilistic Forecasting: Unlike models that output single values, Lag-Llama outputs distribution parameters:

  • For Student-t: degrees of freedom (nu), location (mu), scale (sigma)
  • Allows uncertainty quantification: “The forecast is 100 ± 15”
  • Enables risk-aware decisions: “There’s a 5% chance it exceeds 130”

Why Student-t Distribution?

  • Has heavier tails than Normal (better for extreme events)
  • Degrades gracefully to Normal as nu → infinity
  • More robust to outliers in training data

Zero-Shot Capability: Pre-trained on diverse time series, Lag-Llama can forecast new series without training.

How It Works

Lag-Llama adapts the Llama large language model architecture for time series forecasting. It uses lag-based features to capture temporal patterns at multiple scales and outputs probabilistic forecasts via distribution parameter prediction.

Reference: Rasul et al., “Lag-Llama: Towards Foundation Models for Probabilistic Time Series Forecasting”, 2024. https://arxiv.org/abs/2310.08278

Example

// Create a Lag-Llama for probabilistic time series forecasting
// Adapts the Llama LLM architecture with lag-based features for multi-scale temporal patterns
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 512, inputWidth: 1, inputDepth: 1, outputSize: 24);
            
// Training mode with RoPE, SwiGLU, and Student-t distribution output
var model = new LagLlama<double>(architecture);
            
// ONNX inference mode with pre-trained model
var onnxModel = new LagLlama<double>(architecture, "lag_llama.onnx");

Constructors

Constructor Summary
LagLlama(NeuralNetworkArchitecture<T> architecture, LagLlamaOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Lag-Llama model in native mode for training or fine-tuning.
LagLlama(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, LagLlamaOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Creates a Lag-Llama model using pretrained ONNX model.

Properties

Property Summary
bool IsChannelIndependent { get; }
bool SupportsTraining { get; }
bool UseNativeMode { get; }
int NumFeatures { get; }
int PatchSize { get; }
int PredictionHorizon { get; }
int SequenceLength { get; }
int Stride { get; }
int[] LagIndices { get; } Gets the lag indices used for feature extraction.
string DistributionOutput { get; } Gets the distribution type used for probabilistic output.

Methods

Method Summary
Dictionary<string, T> Evaluate(Tensor<T> predictions, Tensor<T> actuals)
Dictionary<string, T> GetFinancialMetrics()
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata()
Tensor<T> ApplyInstanceNormalization(Tensor<T> input)
Tensor<T> AutoregressiveForecast(Tensor<T> input, int steps)