TimeSeriesTokenizer<T>
Models & Types · AiDotNet.Finance.Tokenization
Standard tokenization pipeline for time series foundation models, supporting patching, quantization, and instance normalization strategies.
For Beginners
Before feeding raw time series data into a foundation model, we need to “tokenize” it — convert it into a format the model expects. Different models use different tokenization strategies:
- Patching (PatchTST, Chronos-2, Moirai): Splits the series into fixed-size chunks
- Quantization (Chronos v1): Maps continuous values to discrete vocabulary tokens
- Instance Normalization (RevIN): Normalizes each series independently
- Adaptive (Kairos): Variable-size patches based on local information density
How It Works
Reference: Standard tokenization approaches from Chronos (ICML 2024), PatchTST (ICLR 2023), and Kairos (2025).
Constructors
| Constructor | Summary |
|---|---|
TimeSeriesTokenizer(TimeSeriesTokenizationStrategy strategy, int patchLength, int? stride, int vocabularySize) |
Creates a new tokenizer with the specified strategy. |
Properties
| Property | Summary |
|---|---|
TimeSeriesTokenizationStrategy Strategy { get; } |
Gets the tokenization strategy being used. |
int PatchLength { get; } |
Gets the patch length for patch-based tokenization. |
int Stride { get; } |
Gets the stride between patches. |
int VocabularySize { get; } |
Gets the vocabulary size for quantization-based tokenization. |
Methods
| Method | Summary |
|---|---|
List<Tensor<T>> CreatePatches(Tensor<T> series) |
Tokenizes a time series into patches. |
Tensor<T> Dequantize(int[] tokens, Tensor<T> binEdges) |
Dequantizes token indices back to continuous values using bin centers. |
Tensor<T> ReconstructFromPatches(List<Tensor<T>> patches) |
Reconstructs a time series from patches (inverse of CreatePatches for non-overlapping case). |
Tensor<T> ReverseInstanceNorm(Tensor<T> prediction, T mean, T std) |
Reverses instance normalization on a prediction. |
ValueTuple<Tensor<T>, T, T> ApplyInstanceNorm(Tensor<T> series) |
Applies instance normalization (RevIN) to a time series. |
ValueTuple<int[], Tensor<T>> Quantize(Tensor<T> series) |
Quantizes continuous values into discrete tokens using uniform binning. |