WikiText2DataLoaderOptions

Options & Configuration · AiDotNet.Data.Text.Benchmarks

Configuration options for the WikiText-2 data loader.

How It Works

WikiText-2 is the small-scale word-level language modeling dataset (≈ 2M tokens train, ≈ 245k tokens validation, ≈ 281k tokens test) drawn from verified Good and Featured Wikipedia articles. It’s the canonical fast-iteration benchmark in the LM literature — papers commonly report WikiText-2 perplexity first (cheap), then WikiText-103 second (slow, 50× larger).

Defaults are tuned for the smaller corpus: SequenceLength=32 keeps memory footprint modest for fast research iteration, VocabularySize=16000 covers roughly the 95th-percentile token-frequency cutoff at this scale. Override these for paper-grade evaluation runs.

Constructors

Constructor Summary
WikiText2DataLoaderOptions()

Properties

Property Summary
DatasetSplit Split { get; set; } Dataset split to load.
bool AutoDownload { get; set; } Automatically download if not present.
int SequenceLength { get; set; } Sequence length for language modeling (context window).
int VocabularySize { get; set; } Maximum vocabulary size.
int? MaxSamples { get; set; } Optional maximum number of samples to load.
string DataPath { get; set; } Root data path.