CsvStreamingDataLoader<T, TInput, TOutput>

Models & Types · AiDotNet.Data.Loaders

A streaming data loader that reads from a CSV file line by line.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput> → CsvStreamingDataLoader<T, TInput, TOutput>

For Beginners

If you have a large CSV file (gigabytes of data), this loader will read it row by row as needed during training.

Example:

How It Works

CsvStreamingDataLoader reads a CSV file line by line without loading the entire file into memory. This is ideal for large tabular datasets.

Constructors

Constructor Summary
CsvStreamingDataLoader(string filePath, Func<string, int, ValueTuple<TInput, TOutput>> lineParser, int batchSize, bool hasHeader, int prefetchCount, int numWorkers) Initializes a new instance of the CsvStreamingDataLoader class.

Properties

Property Summary
int SampleCount { get; }
string Name { get; }

Methods

Method Summary
IEnumerable<ValueTuple<TInput[], TOutput[]>> GetSequentialBatches(int? batchSize, bool dropLast) Iterates through the CSV file sequentially without loading all lines into memory.