Enwik8DataLoader<T>
Models & Types · AiDotNet.Data.Text.Benchmarks
Loads the enwik8 character-level Wikipedia language modeling benchmark (first 100M bytes of an English Wikipedia XML dump).
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → Enwik8DataLoader<T>
How It Works
Expects a single file {DataPath}/enwik8. Auto-download fetches the
canonical zip from mattmahoney.net. Operates byte-by-byte (no tokenization).
Standard split: first 90M chars → train, next 5M → val, last 5M → test.
Features are byte-id sequences Tensor[N, SequenceLength]; labels are
next-byte targets shifted by 1.
Constructors
| Constructor | Summary |
|---|---|
Enwik8DataLoader(Enwik8DataLoaderOptions options) |
Creates a new enwik8 data loader. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
Task<string> LoadRawTextAsync(DatasetSplit split, CancellationToken cancellationToken) |
Loads the raw, unprocessed UTF-8 text content for the requested enwik8 split, auto-downloading via AutoDownload if the file is not already cached. |
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |