TokenizedTextDataset<T>

Models & Types · AiDotNet.Data.Text

In-memory dataset of pre-tokenized text sequences for language model training.

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → TokenizedTextDataset<T>

For Beginners

After tokenizing your text (converting words to numbers), use this dataset to hold the token sequences for training:

How It Works

Stores token ID sequences where each sample is a fixed-length array of integer token IDs. This is the standard input format for transformer-based language models (GPT, BERT, etc.).

Constructors

Constructor Summary
TokenizedTextDataset(int[][] tokenIds, int[] labels, int sequenceLength, int paddingTokenId) Creates a tokenized text dataset from token ID sequences and labels.

Properties

Property Summary
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)