StreamingTextDatasetOptions
Options & Configuration · AiDotNet.Data.Text
Configuration options for the streaming text dataset.
How It Works
Streaming text datasets load text data lazily from disk, enabling training on datasets larger than memory (e.g., The Pile, RedPajama, C4).
Constructors
| Constructor | Summary |
|---|---|
StreamingTextDatasetOptions() |
Properties
| Property | Summary |
|---|---|
bool ShuffleFiles { get; set; } |
Shuffle files before reading. |
int SequenceLength { get; set; } |
Sequence length for each sample. |
int VocabularySize { get; set; } |
Vocabulary size for token ID bounds checking. |
int? MaxSamples { get; set; } |
Optional maximum number of samples to produce. |
int? Seed { get; set; } |
Random seed for shuffling. |
string DataPath { get; set; } |
Root data path containing text files. |
string FilePattern { get; set; } |
File pattern to match. |