WikiText103DataLoaderOptions

Options & Configuration · AiDotNet.Data.Text.Benchmarks

Configuration options for the WikiText-103 data loader.

How It Works

WikiText-103 is a large word-level language modeling dataset with over 100M tokens from verified Good and Featured Wikipedia articles. Used for language model training/evaluation.

Constructors

Constructor Summary
WikiText103DataLoaderOptions()

Properties

Property Summary
DatasetSplit Split { get; set; } Dataset split to load.
bool AutoDownload { get; set; } Automatically download if not present.
int SequenceLength { get; set; } Sequence length for language modeling (context window).
int VocabularySize { get; set; } Maximum vocabulary size.
int? MaxSamples { get; set; } Optional maximum number of samples to load.
string DataPath { get; set; } Root data path.