RWKVForecaster<T>

Models & Types · AiDotNet.Finance.Forecasting.StateSpace

RWKV (Receptance Weighted Key Value) implementation for time series forecasting.

Inheritance: NeuralNetworkBase<T> → FinancialModelBase<T> → ForecastingModelBase<T> → RWKVForecaster<T>

For Beginners

RWKV is a unique architecture for sequence modeling:

The Key Innovation: RWKV can be computed two ways:

  1. As a parallel operation during training (like a Transformer) — fast on GPUs
  2. As a recurrence during inference (like an RNN) — constant memory per token

Architecture Components:

  • Time mixing: WKV (Weighted Key Value) attention with learned exponential decay
  • Channel mixing: Feed-forward network with gating mechanism
  • Token shift: Efficiently mixes current and previous token information

For Time Series:

  • Linear complexity enables processing very long historical windows
  • Constant memory during autoregressive forecasting
  • Multi-head structure captures diverse temporal patterns

How It Works

RWKV combines the efficient parallelizable training of Transformers with the efficient inference of RNNs, achieving linear complexity for both training and inference. This forecasting variant uses stacked RWKV layers for time series prediction.

Reference: Peng et al., “RWKV: Reinventing RNNs for the Transformer Era”, 2023.

Example

var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.Regression,
    inputHeight: 512, inputWidth: 1, inputDepth: 1, outputSize: 24);
var model = new RWKVForecaster<double>(architecture);
var onnxModel = new RWKVForecaster<double>(architecture, "rwkv.onnx");

Constructors

Constructor Summary
RWKVForecaster(NeuralNetworkArchitecture<T> architecture, RWKVForecastingOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance in native mode for training.
RWKVForecaster(NeuralNetworkArchitecture<T> architecture, string onnxModelPath, RWKVForecastingOptions<T> options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer, ILossFunction<T> lossFunction) Initializes a new instance using an ONNX pretrained model.

Properties

Property Summary
bool IsChannelIndependent { get; }
bool SupportsTraining { get; }
bool UseNativeMode { get; }
int NumFeatures { get; }
int NumHeads { get; } Gets the number of RWKV heads.
int PatchSize { get; }
int PredictionHorizon { get; }
int SequenceLength { get; }
int Stride { get; }

Methods

Method Summary
Dictionary<string, T> Evaluate(Tensor<T> predictions, Tensor<T> actuals)
Dictionary<string, T> GetFinancialMetrics()
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input) Captures the per-layer activations along the model’s real forward path.
ModelMetadata<T> GetModelMetadata()
Tensor<T> ApplyInstanceNormalization(Tensor<T> input)
Tensor<T> AutoregressiveForecast(Tensor<T> input, int steps)