StreamingDataLoader<T, TInput, TOutput>
Models & Types · AiDotNet.Data.Loaders
A data loader that streams data from disk or other sources without loading all data into memory.
Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput> → StreamingDataLoader<T, TInput, TOutput>
For Beginners
When your dataset is too large to fit in RAM (e.g., millions of images or text documents), you can’t load it all at once. StreamingDataLoader solves this by reading data piece by piece as needed.
Example:
How It Works
StreamingDataLoader is designed for datasets that don’t fit in memory. Instead of loading all data upfront, it reads data on-demand from a source, processes it, and yields batches.
Constructors
| Constructor | Summary |
|---|---|
StreamingDataLoader(int sampleCount, Func<int, CancellationToken, Task<ValueTuple<TInput, TOutput>>> sampleReader, int batchSize, string name, int prefetchCount, int numWorkers) |
Initializes a new instance of the StreamingDataLoader class. |
Properties
| Property | Summary |
|---|---|
int SampleCount { get; } |
|
string Name { get; } |