PennTreebankDataLoaderOptions

Options & Configuration · AiDotNet.Data.Text.Benchmarks

Configuration options for the Penn Treebank (PTB) data loader.

How It Works

PTB is the classic small-scale word-level LM benchmark — ≈ 887k tokens train, ≈ 70k val, ≈ 79k test, vocab ≈ 10k after preprocessing. Most pre-2018 LM papers report PTB perplexity. Useful as a sanity check before scaling. Tokenization follows the Mikolov-style preprocessed split.

Constructors

Constructor Summary
PennTreebankDataLoaderOptions()

Properties

Property Summary
DatasetSplit Split { get; set; } Dataset split to load.
bool AutoDownload { get; set; } Automatically download if not present.
int SequenceLength { get; set; } Sequence length (BPTT context).
int VocabularySize { get; set; } Maximum vocabulary size.
int? MaxSamples { get; set; } Optional maximum number of samples to load.
string DataPath { get; set; } Root data path.