WebDatasetDataLoader<T>

Models & Types · AiDotNet.Data.Formats

A typed data loader facade that wraps WebDataset and implements StreamingDataLoaderBase for IDataLoader compliance.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, Tensor<T>, Tensor<T>> → WebDatasetDataLoader<T>

For Beginners

Use this when you want to load WebDataset TAR archives and feed them into AiModelBuilder.ConfigureDataLoader(). You provide a parser function that knows how to decode the raw bytes (images, labels, etc.) into tensors.

How It Works

This loader eagerly reads all samples from the underlying TAR archives during loading, caches them in memory, and provides index-based access via a user-supplied parser delegate that converts raw Dictionary<string, byte[]> samples into typed tensor pairs.

Constructors

Constructor Summary
WebDatasetDataLoader(string[] tarPaths, Func<Dictionary<string, byte[]>, ValueTuple<Tensor<T>, Tensor<T>>> sampleParser, int batchSize, WebDatasetOptions options) Creates a new WebDatasetDataLoader.

Properties

Property Summary
int SampleCount { get; }
string Name { get; }