WikiText2DataLoaderOptions
Options & Configuration · AiDotNet.Data.Text.Benchmarks
Configuration options for the WikiText-2 data loader.
How It Works
WikiText-2 is the small-scale word-level language modeling dataset (≈ 2M tokens train, ≈ 245k tokens validation, ≈ 281k tokens test) drawn from verified Good and Featured Wikipedia articles. It’s the canonical fast-iteration benchmark in the LM literature — papers commonly report WikiText-2 perplexity first (cheap), then WikiText-103 second (slow, 50× larger).
Defaults are tuned for the smaller corpus: SequenceLength=32 keeps memory footprint modest for fast research iteration, VocabularySize=16000 covers roughly the 95th-percentile token-frequency cutoff at this scale. Override these for paper-grade evaluation runs.
Constructors
| Constructor | Summary |
|---|---|
WikiText2DataLoaderOptions() |
Properties
| Property | Summary |
|---|---|
DatasetSplit Split { get; set; } |
Dataset split to load. |
bool AutoDownload { get; set; } |
Automatically download if not present. |
int SequenceLength { get; set; } |
Sequence length for language modeling (context window). |
int VocabularySize { get; set; } |
Maximum vocabulary size. |
int? MaxSamples { get; set; } |
Optional maximum number of samples to load. |
string DataPath { get; set; } |
Root data path. |