StreamingTextDataset<T>

Models & Types · AiDotNet.Data.Text

A streaming text dataset that lazily reads and tokenizes text files for language model training.

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → StreamingTextDataset<T>

How It Works

Reads text files sequentially, splits into fixed-length token sequences using a simple word-level tokenizer. Input is tokens[0..n-1], target is tokens[1..n] (next-token prediction). Suitable for pre-training language models on large text corpora.

Constructors

Constructor Summary
StreamingTextDataset(StreamingTextDatasetOptions options)

Properties

Property Summary
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)