Enwik8DataLoaderOptions
Options & Configuration ยท AiDotNet.Data.Text.Benchmarks
Configuration options for the enwik8 character-level LM data loader.
How It Works
enwik8 is the standard character-level Wikipedia language modeling benchmark (Hutter Prize): the first 100M bytes of an English Wikipedia XML dump. Models are evaluated in bits-per-character (BPC); SOTA values typically sit in the 0.95โ1.10 range. Canonical split: first 90M chars train, next 5M val, last 5M test.
Constructors
| Constructor | Summary |
|---|---|
Enwik8DataLoaderOptions() |
Properties
| Property | Summary |
|---|---|
DatasetSplit Split { get; set; } |
Dataset split to load. |
bool AutoDownload { get; set; } |
Automatically download if not present. |
int SequenceLength { get; set; } |
Sequence length (characters per sample). |
int VocabularySize { get; set; } |
Vocabulary size cap. |
int? MaxSamples { get; set; } |
Optional maximum number of samples to load. |
string DataPath { get; set; } |
Root data path. |