WebDatasetDataLoader<T>
Models & Types · AiDotNet.Data.Formats
A typed data loader facade that wraps WebDataset and implements
StreamingDataLoaderBase for IDataLoader compliance.
Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, Tensor<T>, Tensor<T>> → WebDatasetDataLoader<T>
For Beginners
Use this when you want to load WebDataset TAR archives and
feed them into AiModelBuilder.ConfigureDataLoader(). You provide a parser function
that knows how to decode the raw bytes (images, labels, etc.) into tensors.
How It Works
This loader eagerly reads all samples from the underlying TAR archives during loading,
caches them in memory, and provides index-based access via a user-supplied parser delegate
that converts raw Dictionary<string, byte[]> samples into typed tensor pairs.
Constructors
| Constructor | Summary |
|---|---|
WebDatasetDataLoader(string[] tarPaths, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) |
Creates a new WebDatasetDataLoader. |
Properties
| Property | Summary |
|---|---|
int SampleCount { get; } |
|
string Name { get; } |