StreamingDataLoader<T, TInput, TOutput>

Models & Types · AiDotNet.Data.Loaders

A data loader that streams data from disk or other sources without loading all data into memory.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput> → StreamingDataLoader<T, TInput, TOutput>

For Beginners

When your dataset is too large to fit in RAM (e.g., millions of images or text documents), you can’t load it all at once. StreamingDataLoader solves this by reading data piece by piece as needed.

Example:

How It Works

StreamingDataLoader is designed for datasets that don’t fit in memory. Instead of loading all data upfront, it reads data on-demand from a source, processes it, and yields batches.

Constructors

Constructor Summary
StreamingDataLoader(int sampleCount, Func<int, CancellationToken, Task<ValueTuple<TInput, TOutput>>> sampleReader, int batchSize, string name, int prefetchCount, int numWorkers) Initializes a new instance of the StreamingDataLoader class.

Properties

Property Summary
int SampleCount { get; }
string Name { get; }