LmdbDataset<T>

Models & Types · AiDotNet.Data.Formats

Provides read-only access to datasets stored in a custom binary key-value format inspired by LMDB (Lightning Memory-Mapped Database).

Implements: IDisposable

How It Works

Important: This is NOT a native LMDB implementation and is NOT compatible with LMDB databases created by other tools. It is a custom binary key-value store format designed for efficient sequential and random access to ML training data within AiDotNet. For native LMDB interop, use the LightningDB NuGet package.

The on-disk format is a simple binary key-value store:

  • Header: [magic: 4 bytes “LMDB”] [version: 4 bytes] [numEntries: 4 bytes] [indexOffset: 8 bytes]
  • Data section: sequential key-value pairs [keyLen: 4][key: bytes][valueLen: 4][value: bytes]
  • Index section: [offset: 8 bytes] per entry, pointing to start of each key-value pair

Constructors

Constructor Summary
LmdbDataset(LmdbDatasetOptions options) Opens an LMDB dataset from a directory containing a data.mdb file.

Properties

Property Summary
IReadOnlyList<string> Keys { get; } Gets all keys in the dataset.
int Count { get; } Gets the number of entries in the dataset.

Methods

Method Summary
T[] GetAsArray(string key) Gets a value as a double array (assumes 8-byte doubles stored sequentially).
byte[] Get(string key) Gets a raw byte value by key.
byte[] GetByIndex(int index) Gets a value by integer index.
string GetAsString(string key) Gets a value as a string.
void Dispose()
void WriteDataset(string dataPath, IReadOnlyList<KeyValuePair<string, byte[]>> entries) Writes a dataset to LMDB format.