Enwik8DataLoaderOptions

Options & Configuration ยท AiDotNet.Data.Text.Benchmarks

Configuration options for the enwik8 character-level LM data loader.

How It Works

enwik8 is the standard character-level Wikipedia language modeling benchmark (Hutter Prize): the first 100M bytes of an English Wikipedia XML dump. Models are evaluated in bits-per-character (BPC); SOTA values typically sit in the 0.95โ€“1.10 range. Canonical split: first 90M chars train, next 5M val, last 5M test.

Constructors

Constructor Summary
Enwik8DataLoaderOptions()

Properties

Property Summary
DatasetSplit Split { get; set; } Dataset split to load.
bool AutoDownload { get; set; } Automatically download if not present.
int SequenceLength { get; set; } Sequence length (characters per sample).
int VocabularySize { get; set; } Vocabulary size cap.
int? MaxSamples { get; set; } Optional maximum number of samples to load.
string DataPath { get; set; } Root data path.