RetryChatMiddleware
Models & Types · AiDotNet.Agentic.Pipeline
Retries transient chat failures with bounded exponential backoff, jitter, and a per-call timeout.
Implements: IChatMiddleware
For Beginners
Network calls to AI providers fail sometimes — a connection drops, the service is briefly overloaded, a request takes too long. This wrapper quietly tries again a few times, waiting a little longer between attempts, and gives up with a clear error if the problem persists. It deliberately does not retry mistakes that will never succeed, such as a wrong API key. Add it once and every model call in your application gets the same behaviour.
How It Works
Retry lives here, in the pipeline, rather than inside one connector base class. That single change makes the policy apply to every backend the library can talk to — a local engine, an Ollama endpoint, a Microsoft Extensions AI bridge, a recorded replay client — instead of only to the HTTP connectors that happen to derive from a particular base. It also makes the policy configurable and observable, which a protected field on a base class is not.
A call is attempted at most MaxRetries + 1 times, and that is a guarantee rather than an
approximation. The reference OpenEvolve implementation configures retries in its own loop and passes
the same number to the provider SDK’s internal retry, so a configured value of three can produce up to sixteen
HTTP attempts against a rate-limited endpoint — the exact circumstance where retrying hardest is most harmful.
Delay grows exponentially from the base delay, is capped, and carries jitter drawn from a seeded
StableRandom, so a fleet of workers that all fail at once does not resynchronize on the retry.
Only transient conditions are retried: transport failures, request timeouts, and HTTP 408, 429, and 5xx. A rejected key or a malformed request is returned immediately, because retrying it wastes the budget and delays the error the caller needs to see. Caller cancellation is never treated as a failure.
Constructors
| Constructor | Summary |
|---|---|
RetryChatMiddleware(int maxRetries, TimeSpan? baseDelay, TimeSpan? maxDelay, TimeSpan? callTimeout, double jitterFactor, ulong seed, Func<Exception, bool> shouldRetry) |
Initializes a retry policy. |
Properties
| Property | Summary |
|---|---|
TimeSpan BaseDelay { get; } |
Gets the wait before the first retry. |
TimeSpan MaxDelay { get; } |
Gets the ceiling on any single wait. |
TimeSpan? CallTimeout { get; } |
Gets the time limit for one attempt, or null when attempts are not time-limited. |
double JitterFactor { get; } |
Gets the fraction of each wait that is randomized. |
int MaxRetries { get; } |
Gets the number of retries attempted after the first attempt. |
long TotalAttempts { get; } |
Gets how many attempts, including first attempts, this policy has made. |
long TotalRetries { get; } |
Gets how many of those attempts were retries. |
long TotalTimeouts { get; } |
Gets how many attempts exceeded CallTimeout. |
Methods
| Method | Summary |
|---|---|
Task<ChatResponse> InvokeAsync(ChatRequestContext context, ChatPipelineDelegate next, CancellationToken cancellationToken) |
|
bool IsTransient(Exception exception) |
Classifies an exception as a transient condition worth retrying. |
Fields
| Field | Summary |
|---|---|
int DefaultMaxRetries |
The default number of retries after the first attempt. |
int MaxSupportedRetries |
The largest number of retries accepted. |