CachingEmbeddingClient

Models & Types · AiDotNet.Agentic.Embeddings

Caches embedding vectors by content hash so a text already embedded costs nothing to embed again.

Implements: IEmbeddingClient

For Beginners

Asking a provider to embed text costs money and time, and evolutionary search asks about the same text over and over. This wrapper remembers answers it has already received, keyed by a fingerprint of the text, and only asks about text it has not seen. Wrap any client in it and the rest of your code does not change; Hits and InnerCalls tell you how much it saved.

How It Works

The key is the SHA-256 of the text, so two candidates that are byte-identical share one vector no matter which batch they arrived in, and a candidate re-proposed later in a run never reaches the provider a second time. Only the misses are forwarded: a batch of ten texts of which nine are cached becomes one request for one text, and a batch with no misses becomes no request at all. That is the whole point — an evolutionary search re-proposes near-identical programs constantly, and the reference implementation pays a provider call for every one of them because it embeds inside the insertion path with no cache anywhere.

A failed batch is never cached. Upstream stores its failure sentinel on the program record permanently, so one transient error keeps that record’s similarity at zero for the rest of the run; here the next call simply tries again. Eviction is first-in-first-out at Capacity entries, which bounds memory without the bookkeeping of a recency order, and the cache is safe for concurrent use.

Constructors

Constructor Summary
CachingEmbeddingClient(IEmbeddingClient inner, int capacity) Initializes a caching decorator around another client.

Properties

Property Summary
IEmbeddingClient Inner { get; } Gets the client that answers cache misses.
int Capacity { get; } Gets the number of vectors retained before the oldest is evicted.
int Count { get; } Gets how many vectors the cache currently holds.
long Hits { get; } Gets how many requested texts were served from the cache.
long InnerCalls { get; } Gets how many batches were forwarded to the inner client.
long Misses { get; } Gets how many requested texts had to be embedded.
string ModelId { get; }

Methods

Method Summary
ValueTask<EmbeddingBatch> EmbedAsync(IReadOnlyList<string> texts, CancellationToken cancellationToken)
string ComputeKey(string text) Computes the cache key for a text.
void Clear() Removes every cached vector, leaving the counters untouched.

Fields

Field Summary
int DefaultCapacity The default number of vectors retained.