TokenizedTextDataset<T>
Models & Types · AiDotNet.Data.Text
In-memory dataset of pre-tokenized text sequences for language model training.
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → TokenizedTextDataset<T>
For Beginners
After tokenizing your text (converting words to numbers), use this dataset to hold the token sequences for training:
How It Works
Stores token ID sequences where each sample is a fixed-length array of integer token IDs. This is the standard input format for transformer-based language models (GPT, BERT, etc.).
Constructors
| Constructor | Summary |
|---|---|
TokenizedTextDataset(int[][] tokenIds, int[] labels, int sequenceLength, int paddingTokenId) |
Creates a tokenized text dataset from token ID sequences and labels. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |