StreamingTextDatasetOptions

Options & Configuration · AiDotNet.Data.Text

Configuration options for the streaming text dataset.

How It Works

Streaming text datasets load text data lazily from disk, enabling training on datasets larger than memory (e.g., The Pile, RedPajama, C4).

Constructors

Constructor Summary
StreamingTextDatasetOptions()

Properties

Property Summary
bool ShuffleFiles { get; set; } Shuffle files before reading.
int SequenceLength { get; set; } Sequence length for each sample.
int VocabularySize { get; set; } Vocabulary size for token ID bounds checking.
int? MaxSamples { get; set; } Optional maximum number of samples to produce.
int? Seed { get; set; } Random seed for shuffling.
string DataPath { get; set; } Root data path containing text files.
string FilePattern { get; set; } File pattern to match.