FileDocumentStore<T>
Models & Types ยท AiDotNet.RetrievalAugmentedGeneration.DocumentStores
A durable, file-based vector document store with HNSW indexing and write-ahead logging.
Inheritance: DocumentStoreBase<T> โ FileDocumentStore<T>
Implements: IDisposable
For Beginners
This is like a persistent library that survives application restarts.
Key features:
- Data is saved to disk files so it survives crashes and restarts
- Fast similarity search using HNSW graph index (same as the in-memory store)
- Write-ahead log ensures no data loss even during crashes
- Deleted documents are marked (not immediately removed) and cleaned up periodically
File layout on disk:
Best used for:
- Applications that need persistent vector search
- Medium datasets (up to ~1M documents depending on RAM)
- Single-process applications (not for multi-process concurrent access)
- Scenarios where you want an embedded vector database without external dependencies
Thread-safety: While ConcurrentDictionary is used
internally for document/metadata storage, HNSW index operations (AddVectors, RemoveVectors,
Search/Query) are not protected by internal locks. Concurrent mutations from multiple
threads are unsupported. Concurrent read-only/search operations may work depending on the
HNSW implementation but are not guaranteed. For safe in-process concurrency, serialize writes
with an external lock or implement internal synchronization around HNSW accesses.
How It Works
FileDocumentStore persists documents to disk while maintaining an in-memory HNSW index for O(log n) approximate nearest neighbor search. It uses a write-ahead log (WAL) for crash recovery and tombstone-based soft deletes with periodic compaction.
Constructors
| Constructor | Summary |
|---|---|
FileDocumentStore(int vectorDimension, FileDocumentStoreOptions options) |
Initializes a new FileDocumentStore, creating or loading from the specified directory. |
Properties
| Property | Summary |
|---|---|
int DocumentCount { get; } |
|
int TombstoneCount { get; } |
Gets the number of tombstoned (soft-deleted) documents awaiting compaction. |
int VectorDimension { get; } |
|
string DirectoryPath { get; } |
Gets the directory path where store files are located. |
Methods
| Method | Summary |
|---|---|
void Clear() |
Removes all documents from the store and deletes all files. |
void Compact() |
Performs compaction: rebuilds store files removing tombstoned entries and reclaiming space. |
void Dispose() |
Releases resources used by the store. |
void Flush() |
Flushes all in-memory data to disk, creating a consistent snapshot. |