MemoryMappedDataset<T>

Models & Types · AiDotNet.Data.Formats

Memory-mapped dataset access for efficient I/O on large binary datasets.

Implements: IDisposable

For Beginners

Use this for very large datasets that don’t fit in RAM. The operating system will transparently page data in and out as needed.

How It Works

Memory-mapped files allow the operating system to map file data directly into virtual memory, enabling efficient access to large datasets without loading them entirely into RAM. The OS handles paging data in and out as needed.

The dataset file uses a simple binary format:

  • First 4 bytes: number of samples (int32 little-endian)
  • Next 4 bytes: elements per sample (int32 little-endian)
  • Remaining bytes: raw double-precision (8-byte) values, converted to/from T at read/write time.

When reading, this class materializes samples into managed arrays/tensors (not zero-copy), but the OS-level memory mapping avoids reading the entire file into RAM upfront.

Constructors

Constructor Summary
MemoryMappedDataset(string filePath) Creates a new memory-mapped dataset from a file.

Properties

Property Summary
int ElementsPerSample { get; } Gets the number of elements per sample.
int NumSamples { get; } Gets the total number of samples in the dataset.

Methods

Method Summary
IEnumerable<T[]> EnumerateSamples() Creates an iterator that yields all samples in sequential order.
T[] ReadSample(int index) Reads a single sample at the specified index.
Tensor<T> ReadBatch(int[] indices, int[] sampleShape) Reads a batch of samples at the specified indices.
Tensor<T> ReadSampleAsTensor(int index, int[] shape) Reads a single sample as a Tensor.
void Dispose()
void WriteDatasetFile(string filePath, Tensor<T> data) Writes a dataset file in the expected format.