StreamingDataLoaderBase<T, TInput, TOutput>
Base Classes · AiDotNet.Data.Loaders
Abstract base class for streaming data loaders that process data on-demand.
Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput>
Implements: IStreamingDataLoader<T, TInput, TOutput>
Known derived types: CsvStreamingDataLoader<T, TInput, TOutput>, FileStreamingDataLoader<T, TInput, TOutput>, JsonlDataLoader<T>, MemoryMappedStreamingDataLoader<T, TInput, TOutput>, ShardedStreamingDataLoader<T>, StreamingDataLoader<T, TInput, TOutput>, WebDatasetDataLoader<T>
For Beginners
When working with huge datasets (millions of images, terabytes of text), you can’t load everything into memory at once. This base class handles the complexity of streaming data efficiently while you focus on implementing the actual data reading logic.
How It Works
StreamingDataLoaderBase provides the foundation for data loaders that read data on-demand rather than loading everything into memory. This is essential for:
- Large datasets that don’t fit in RAM
- Real-time data streams
- Memory-efficient training pipelines
Properties
| Property | Summary |
|---|---|
int NumWorkers { get; } |
|
int PrefetchCount { get; } |
|
int SampleCount { get; } |
|
int TotalCount { get; } |
Methods
| Method | Summary |
|---|---|
IAsyncEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatchesAsync(bool shuffle, bool dropLast, int? seed, CancellationToken cancellationToken) |
|
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatches(bool shuffle, bool dropLast, int? seed) |