MemoryMappedStreamingDataLoader<T, TInput, TOutput>
Models & Types · AiDotNet.Data.Loaders
A streaming data loader that uses memory-mapped files for efficient random access to large binary datasets.
Inheritance: DataLoaderBase<T> → StreamingDataLoaderBase<T, TInput, TOutput> → MemoryMappedStreamingDataLoader<T, TInput, TOutput>
Implements: IDisposable
For Beginners
Memory-mapped files let the operating system manage which parts of a large file are in memory. When you access a sample, the OS automatically loads that portion of the file into RAM. This is very efficient for random access patterns like shuffled batch iteration on datasets too large to fit in memory.
Example:
How It Works
MemoryMappedStreamingDataLoader uses MemoryMappedFile
for efficient random access to large datasets stored in binary format. The operating system
handles paging data in and out of physical memory as needed, enabling efficient access to
datasets larger than available RAM.
File Format Requirements:
- Binary file with fixed-size samples
- Each sample is
inputSizeBytes + outputSizeBytesbytes - Samples are stored contiguously with optional header
Constructors
| Constructor | Summary |
|---|---|
MemoryMappedStreamingDataLoader(string filePath, int sampleCount, int inputSizeBytes, int outputSizeBytes, Func<byte[], TInput> inputDeserializer, Func<byte[], TOutput> outputDeserializer, int batchSize, long headerSizeBytes, int prefetchCount, int numWorkers) |
Initializes a new instance of the MemoryMappedStreamingDataLoader class. |
Properties
| Property | Summary |
|---|---|
int SampleCount { get; } |
|
int SampleSizeBytes { get; } |
Gets the size of each sample in bytes (input + output). |
long HeaderSizeBytes { get; } |
Gets the size of the file header in bytes. |
string Name { get; } |
Methods
| Method | Summary |
|---|---|
void Dispose() |
Releases all resources used by the memory-mapped data loader. |