PennTreebankDataLoaderOptions
Options & Configuration · AiDotNet.Data.Text.Benchmarks
Configuration options for the Penn Treebank (PTB) data loader.
How It Works
PTB is the classic small-scale word-level LM benchmark — ≈ 887k tokens train, ≈ 70k val, ≈ 79k test, vocab ≈ 10k after preprocessing. Most pre-2018 LM papers report PTB perplexity. Useful as a sanity check before scaling. Tokenization follows the Mikolov-style preprocessed split.
Constructors
| Constructor | Summary |
|---|---|
PennTreebankDataLoaderOptions() |
Properties
| Property | Summary |
|---|---|
DatasetSplit Split { get; set; } |
Dataset split to load. |
bool AutoDownload { get; set; } |
Automatically download if not present. |
int SequenceLength { get; set; } |
Sequence length (BPTT context). |
int VocabularySize { get; set; } |
Maximum vocabulary size. |
int? MaxSamples { get; set; } |
Optional maximum number of samples to load. |
string DataPath { get; set; } |
Root data path. |