IStreamingDataLoader<T, TInput, TOutput>

Interfaces · AiDotNet.Interfaces

Interface for streaming data loaders that process data on-demand without loading all data into memory.

Implements: IDataLoader<T>

Known implementations: DistributedStreamingDataLoader<T, TInput, TOutput>, StreamingDataLoaderBase<T, TInput, TOutput>

For Beginners

When your dataset is too large to fit in RAM (like millions of images or text documents), you can’t load it all at once. Streaming data loaders solve this by reading data piece by piece as needed during training.

Example usage:

How It Works

IStreamingDataLoader is designed for datasets that are too large to fit in memory. Unlike IInputOutputDataLoader which provides Features and Labels properties for all data, streaming loaders read data on-demand and yield batches through iteration.

Properties

Property Summary
int BatchSize { get; set; } Gets or sets the batch size for iteration.
int NumWorkers { get; } Gets the number of parallel workers for sample loading.
int PrefetchCount { get; } Gets the number of batches to prefetch for improved throughput.
int SampleCount { get; } Gets the total number of samples in the dataset.

Methods

Method Summary
IAsyncEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatchesAsync(bool shuffle, bool dropLast, int? seed, CancellationToken cancellationToken) Iterates through the dataset in batches asynchronously with prefetching.
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetBatches(bool shuffle, bool dropLast, int? seed) Iterates through the dataset in batches synchronously.