FileDocumentStore<T>

Models & Types ยท AiDotNet.RetrievalAugmentedGeneration.DocumentStores

A durable, file-based vector document store with HNSW indexing and write-ahead logging.

Inheritance: DocumentStoreBase<T> โ†’ FileDocumentStore<T>

Implements: IDisposable

For Beginners

This is like a persistent library that survives application restarts.

Key features:

  • Data is saved to disk files so it survives crashes and restarts
  • Fast similarity search using HNSW graph index (same as the in-memory store)
  • Write-ahead log ensures no data loss even during crashes
  • Deleted documents are marked (not immediately removed) and cleaned up periodically

File layout on disk:

Best used for:

  • Applications that need persistent vector search
  • Medium datasets (up to ~1M documents depending on RAM)
  • Single-process applications (not for multi-process concurrent access)
  • Scenarios where you want an embedded vector database without external dependencies

Thread-safety: While ConcurrentDictionary is used internally for document/metadata storage, HNSW index operations (AddVectors, RemoveVectors, Search/Query) are not protected by internal locks. Concurrent mutations from multiple threads are unsupported. Concurrent read-only/search operations may work depending on the HNSW implementation but are not guaranteed. For safe in-process concurrency, serialize writes with an external lock or implement internal synchronization around HNSW accesses.

How It Works

FileDocumentStore persists documents to disk while maintaining an in-memory HNSW index for O(log n) approximate nearest neighbor search. It uses a write-ahead log (WAL) for crash recovery and tombstone-based soft deletes with periodic compaction.

Constructors

Constructor Summary
FileDocumentStore(int vectorDimension, FileDocumentStoreOptions options) Initializes a new FileDocumentStore, creating or loading from the specified directory.

Properties

Property Summary
int DocumentCount { get; }
int TombstoneCount { get; } Gets the number of tombstoned (soft-deleted) documents awaiting compaction.
int VectorDimension { get; }
string DirectoryPath { get; } Gets the directory path where store files are located.

Methods

Method Summary
void Clear() Removes all documents from the store and deletes all files.
void Compact() Performs compaction: rebuilds store files removing tombstoned entries and reclaiming space.
void Dispose() Releases resources used by the store.
void Flush() Flushes all in-memory data to disk, creating a consistent snapshot.