StreamingTextDataset<T>
Models & Types · AiDotNet.Data.Text
A streaming text dataset that lazily reads and tokenizes text files for language model training.
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → StreamingTextDataset<T>
How It Works
Reads text files sequentially, splits into fixed-length token sequences using a simple word-level tokenizer. Input is tokens[0..n-1], target is tokens[1..n] (next-token prediction). Suitable for pre-training language models on large text corpora.
Constructors
| Constructor | Summary |
|---|---|
StreamingTextDataset(StreamingTextDatasetOptions options) |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |