CacheConfig

Options & Configuration · AiDotNet.Deployment.Configuration

Configuration for model caching - storing loaded models in memory to avoid repeated loading.

For Beginners

Loading an AI model from disk takes time. Caching keeps recently-used models in memory so they can be used again instantly, like keeping your frequently-used apps open on your phone instead of closing and reopening them.

Benefits:

  • Much faster inference (no model loading time)
  • Better throughput when serving multiple requests
  • Reduces disk I/O

Considerations:

  • Uses memory (RAM) to store models
  • Limited cache size - old models get evicted when full

Eviction Policies (what to remove when cache is full):

  • LRU (Least Recently Used): Removes models you haven’t used in a while (recommended)
  • LFU (Least Frequently Used): Removes models used least often
  • FIFO: Removes oldest models first
  • Random: Removes random models (simple but unpredictable)

For most applications, LRU with a moderate max size works well.

Constructors

Constructor Summary
CacheConfig()

Properties

Property Summary
CacheEvictionPolicy EvictionPolicy { get; set; } Gets or sets the cache eviction policy (default: LRU).
CacheEvictionPolicy OptimizerCacheEvictionPolicy { get; set; } Eviction policy for BOTH optimizer caches (default: FIFO).
TimeSpan DefaultTTL { get; set; } Gets or sets the cache entry time-to-live as a TimeSpan (default: 1 hour).
bool Enabled { get; set; } Gets or sets whether caching is enabled (default: true).
bool PreloadModels { get; set; } Gets or sets whether to preload models on startup (default: false).
bool TrackStatistics { get; set; } Gets or sets whether to track cache hit/miss statistics (default: true).
double MaxSizeMB { get; set; } Gets or sets the maximum cache size in megabytes (default: 100.0 MB).
int GradientCacheCapacity { get; set; } Maximum number of per-step gradients retained by the optimizer’s gradient cache (DefaultGradientCache).
int MaxCacheSize { get; set; } Gets or sets the maximum number of models to cache (default: 10).
int ModelCacheCapacity { get; set; } Maximum number of per-evaluation step-data entries retained by the optimizer’s model-evaluation cache (DefaultModelCache).
int TimeToLiveSeconds { get; set; } Gets or sets the cache entry time-to-live in seconds (default: 3600 = 1 hour).