TinyStoriesDataLoader<T>
Models & Types · AiDotNet.Data.Text.Benchmarks
Loads the TinyStories synthetic LM corpus (Eldan & Li, 2023): ≈ 2.1M GPT-generated short stories with deliberately small vocabulary for small-scale language-model research.
Inheritance: DataLoaderBase<T> → InputOutputDataLoaderBase<T, Tensor<T>, Tensor<T>> → TinyStoriesDataLoader<T>
How It Works
Expects:
Auto-download fetches both files from the canonical Hugging Face
repository roneneldan/TinyStories. Stories are concatenated and
chunked into fixed-length input/target sequences for LM training.
Constructors
| Constructor | Summary |
|---|---|
TinyStoriesDataLoader(TinyStoriesDataLoaderOptions options) |
Creates a new TinyStories data loader. |
Properties
| Property | Summary |
|---|---|
int FeatureCount { get; } |
|
int OutputDimension { get; } |
|
int TotalCount { get; } |
|
string Description { get; } |
|
string Name { get; } |
Methods
| Method | Summary |
|---|---|
ValueTuple<IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>, IInputOutputDataLoader<T, Tensor<T>, Tensor<T>>> Split(double trainRatio, double validationRatio, int? seed) |