StreamingDataLoaderBase<T, TInput, TOutput>

Base Classes · AiDotNet.Data.Loaders

Abstract base class for streaming data loaders that process data on-demand.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput>

Implements: IStreamingDataLoader<T, TInput, TOutput>

Known derived types: CsvStreamingDataLoader<T, TInput, TOutput>, FileStreamingDataLoader<T, TInput, TOutput>, JsonlDataLoader<T>, MemoryMappedStreamingDataLoader<T, TInput, TOutput>, ShardedStreamingDataLoader<T>, StreamingDataLoader<T, TInput, TOutput>, WebDatasetDataLoader<T>

For Beginners

When working with huge datasets (millions of images, terabytes of text), you can’t load everything into memory at once. This base class handles the complexity of streaming data efficiently while you focus on implementing the actual data reading logic.

How It Works

StreamingDataLoaderBase provides the foundation for data loaders that read data on-demand rather than loading everything into memory. This is essential for:

  • Large datasets that don’t fit in RAM
  • Real-time data streams
  • Memory-efficient training pipelines

Properties

Property Summary
int NumWorkers { get; }
int PrefetchCount { get; }
int SampleCount { get; }
int TotalCount { get; }

Methods

Method Summary
IAsyncEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatchesAsync(bool shuffle, bool dropLast, int? seed, CancellationToken cancellationToken)
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatches(bool shuffle, bool dropLast, int? seed)