MemoryMappedStreamingDataLoader<T, TInput, TOutput>

Models & Types · AiDotNet.Data.Loaders

A streaming data loader that uses memory-mapped files for efficient random access to large binary datasets.

Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput> → MemoryMappedStreamingDataLoader<T, TInput, TOutput>

Implements: IDisposable

For Beginners

Memory-mapped files let the operating system manage which parts of a large file are in memory. When you access a sample, the OS automatically loads that portion of the file into RAM. This is very efficient for random access patterns like shuffled batch iteration on datasets too large to fit in memory.

Example:

How It Works

MemoryMappedStreamingDataLoader uses MemoryMappedFile for efficient random access to large datasets stored in binary format. The operating system handles paging data in and out of physical memory as needed, enabling efficient access to datasets larger than available RAM.

File Format Requirements:

  • Binary file with fixed-size samples
  • Each sample is inputSizeBytes + outputSizeBytes bytes
  • Samples are stored contiguously with optional header

Constructors

Constructor Summary
MemoryMappedStreamingDataLoader(string filePath, int sampleCount, int inputSizeBytes, int outputSizeBytes, Func<byte[], TInput> inputDeserializer, Func<byte[], TOutput> outputDeserializer, int batchSize, long headerSizeBytes, int prefetchCount, int numWorkers) Initializes a new instance of the MemoryMappedStreamingDataLoader class.

Properties

Property Summary
int SampleCount { get; }
int SampleSizeBytes { get; } Gets the size of each sample in bytes (input + output).
long HeaderSizeBytes { get; } Gets the size of the file header in bytes.
string Name { get; }

Methods

Method Summary
void Dispose() Releases all resources used by the memory-mapped data loader.