PennTreebankDataLoader<T>

Models & Types · AiDotNet.Data.Text.Benchmarks

Loads the Penn Treebank language modeling dataset (Mikolov-preprocessed split).

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → PennTreebankDataLoader<T>

How It Works

Expects:

Auto-download fetches the canonical Mikolov tarball. Features are input token sequences Tensor[N, SequenceLength]. Labels are next-token targets Tensor[N, SequenceLength] (shifted by 1).

Constructors

Constructor Summary
PennTreebankDataLoader(PennTreebankDataLoaderOptions options) Creates a new PTB data loader.

Properties

Property Summary
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
Task<string> LoadRawTextAsync(DatasetSplit split, CancellationToken cancellationToken) Loads the raw, unprocessed text content for the requested PTB split, auto-downloading via AutoDownload if the file is not already cached.
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)