Enwik8DataLoader<T>

Models & Types · AiDotNet.Data.Text.Benchmarks

Loads the enwik8 character-level Wikipedia language modeling benchmark (first 100M bytes of an English Wikipedia XML dump).

Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → Enwik8DataLoader<T>

How It Works

Expects a single file {DataPath}/enwik8. Auto-download fetches the canonical zip from mattmahoney.net. Operates byte-by-byte (no tokenization). Standard split: first 90M chars → train, next 5M → val, last 5M → test. Features are byte-id sequences Tensor[N, SequenceLength]; labels are next-byte targets shifted by 1.

Constructors

Constructor Summary
Enwik8DataLoader(Enwik8DataLoaderOptions options) Creates a new enwik8 data loader.

Properties

Property Summary
int FeatureCount { get; }
int OutputDimension { get; }
int TotalCount { get; }
string Description { get; }
string Name { get; }

Methods

Method Summary
Task<string> LoadRawTextAsync(DatasetSplit split, CancellationToken cancellationToken) Loads the raw, unprocessed UTF-8 text content for the requested enwik8 split, auto-downloading via AutoDownload if the file is not already cached.
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed)