RetryChatMiddleware

Models & Types · AiDotNet.Agentic.Pipeline

Retries transient chat failures with bounded exponential backoff, jitter, and a per-call timeout.

Implements: IChatMiddleware

For Beginners

Network calls to AI providers fail sometimes — a connection drops, the service is briefly overloaded, a request takes too long. This wrapper quietly tries again a few times, waiting a little longer between attempts, and gives up with a clear error if the problem persists. It deliberately does not retry mistakes that will never succeed, such as a wrong API key. Add it once and every model call in your application gets the same behaviour.

How It Works

Retry lives here, in the pipeline, rather than inside one connector base class. That single change makes the policy apply to every backend the library can talk to — a local engine, an Ollama endpoint, a Microsoft Extensions AI bridge, a recorded replay client — instead of only to the HTTP connectors that happen to derive from a particular base. It also makes the policy configurable and observable, which a protected field on a base class is not.

A call is attempted at most MaxRetries + 1 times, and that is a guarantee rather than an approximation. The reference OpenEvolve implementation configures retries in its own loop and passes the same number to the provider SDK’s internal retry, so a configured value of three can produce up to sixteen HTTP attempts against a rate-limited endpoint — the exact circumstance where retrying hardest is most harmful. Delay grows exponentially from the base delay, is capped, and carries jitter drawn from a seeded StableRandom, so a fleet of workers that all fail at once does not resynchronize on the retry.

Only transient conditions are retried: transport failures, request timeouts, and HTTP 408, 429, and 5xx. A rejected key or a malformed request is returned immediately, because retrying it wastes the budget and delays the error the caller needs to see. Caller cancellation is never treated as a failure.

Constructors

Constructor Summary
RetryChatMiddleware(int maxRetries, TimeSpan? baseDelay, TimeSpan? maxDelay, TimeSpan? callTimeout, double jitterFactor, ulong seed, Func<Exception, bool> shouldRetry) Initializes a retry policy.

Properties

Property Summary
TimeSpan BaseDelay { get; } Gets the wait before the first retry.
TimeSpan MaxDelay { get; } Gets the ceiling on any single wait.
TimeSpan? CallTimeout { get; } Gets the time limit for one attempt, or null when attempts are not time-limited.
double JitterFactor { get; } Gets the fraction of each wait that is randomized.
int MaxRetries { get; } Gets the number of retries attempted after the first attempt.
long TotalAttempts { get; } Gets how many attempts, including first attempts, this policy has made.
long TotalRetries { get; } Gets how many of those attempts were retries.
long TotalTimeouts { get; } Gets how many attempts exceeded CallTimeout.

Methods

Method Summary
Task<ChatResponse> InvokeAsync(ChatRequestContext context, ChatPipelineDelegate next, CancellationToken cancellationToken)
bool IsTransient(Exception exception) Classifies an exception as a transient condition worth retrying.

Fields

Field Summary
int DefaultMaxRetries The default number of retries after the first attempt.
int MaxSupportedRetries The largest number of retries accepted.