LmdbDataset<T>
Models & Types · AiDotNet.Data.Formats
Provides read-only access to datasets stored in a custom binary key-value format inspired by LMDB (Lightning Memory-Mapped Database).
Implements: IDisposable
How It Works
Important: This is NOT a native LMDB implementation and is NOT compatible with LMDB databases created by other tools. It is a custom binary key-value store format designed for efficient sequential and random access to ML training data within AiDotNet. For native LMDB interop, use the LightningDB NuGet package.
The on-disk format is a simple binary key-value store:
- Header: [magic: 4 bytes “LMDB”] [version: 4 bytes] [numEntries: 4 bytes] [indexOffset: 8 bytes]
- Data section: sequential key-value pairs [keyLen: 4][key: bytes][valueLen: 4][value: bytes]
- Index section: [offset: 8 bytes] per entry, pointing to start of each key-value pair
Constructors
| Constructor | Summary |
|---|---|
LmdbDataset(LmdbDatasetOptions options) |
Opens an LMDB dataset from a directory containing a data.mdb file. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> Keys { get; } |
Gets all keys in the dataset. |
int Count { get; } |
Gets the number of entries in the dataset. |
Methods
| Method | Summary |
|---|---|
T[] GetAsArray(string key) |
Gets a value as a double array (assumes 8-byte doubles stored sequentially). |
byte[] Get(string key) |
Gets a raw byte value by key. |
byte[] GetByIndex(int index) |
Gets a value by integer index. |
string GetAsString(string key) |
Gets a value as a string. |
void Dispose() |
|
void WriteDataset(string dataPath, IReadOnlyList<KeyValuePair<string, byte[]>> entries) |
Writes a dataset to LMDB format. |