WikiText2DataLoader<T>
Models & Types · AiDotNet.Data.Text.Benchmarks
Loads the WikiText-2 language modeling dataset (≈ 2M tokens train, ≈ 245k val, ≈ 281k test from Wikipedia articles).
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → WikiText2DataLoader<T>
How It Works
WikiText-2 expects:
Features are input token sequences Tensor[N, SequenceLength]. Labels are next-token target sequences Tensor[N, SequenceLength] (shifted by 1).
Uses the wikitext-2-raw-v1 archive flavor (preserves original casing
and punctuation), which is the modern standard for subword-tokenizer
pipelines. The companion WikiText103DataLoader uses the
older tokenized flavor for backward compatibility — pick whichever matches
your tokenizer.
Constructors
| Constructor | Summary |
|---|---|
WikiText2DataLoader(WikiText2DataLoaderOptions options) |
Creates a new WikiText-2 data loader. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
Task<string> LoadRawTextAsync(DatasetSplit split, CancellationToken cancellationToken) |
Loads the raw, unprocessed text content for the requested split, auto-downloading via AutoDownload if the file is not already cached. |
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |