TimeSeriesTokenizer<T>

Models & Types · AiDotNet.Finance.Tokenization

Standard tokenization pipeline for time series foundation models, supporting patching, quantization, and instance normalization strategies.

For Beginners

Before feeding raw time series data into a foundation model, we need to “tokenize” it — convert it into a format the model expects. Different models use different tokenization strategies:

  • Patching (PatchTST, Chronos-2, Moirai): Splits the series into fixed-size chunks
  • Quantization (Chronos v1): Maps continuous values to discrete vocabulary tokens
  • Instance Normalization (RevIN): Normalizes each series independently
  • Adaptive (Kairos): Variable-size patches based on local information density

How It Works

Reference: Standard tokenization approaches from Chronos (ICML 2024), PatchTST (ICLR 2023), and Kairos (2025).

Constructors

Constructor Summary
TimeSeriesTokenizer(TimeSeriesTokenizationStrategy strategy, int patchLength, int? stride, int vocabularySize) Creates a new tokenizer with the specified strategy.

Properties

Property Summary
TimeSeriesTokenizationStrategy Strategy { get; } Gets the tokenization strategy being used.
int PatchLength { get; } Gets the patch length for patch-based tokenization.
int Stride { get; } Gets the stride between patches.
int VocabularySize { get; } Gets the vocabulary size for quantization-based tokenization.

Methods

Method Summary
List<Tensor<T>> CreatePatches(Tensor<T> series) Tokenizes a time series into patches.
Tensor<T> Dequantize(int[] tokens, Tensor<T> binEdges) Dequantizes token indices back to continuous values using bin centers.
Tensor<T> ReconstructFromPatches(List<Tensor<T>> patches) Reconstructs a time series from patches (inverse of CreatePatches for non-overlapping case).
Tensor<T> ReverseInstanceNorm(Tensor<T> prediction, T mean, T std) Reverses instance normalization on a prediction.
ValueTuple<Tensor<T>, T, T> ApplyInstanceNorm(Tensor<T> series) Applies instance normalization (RevIN) to a time series.
ValueTuple<int[], Tensor<T>> Quantize(Tensor<T> series) Quantizes continuous values into discrete tokens using uniform binning.