InferenceOptimizationConfig

Options & Configuration · AiDotNet.Configuration

Configuration for inference-time optimizations to maximize prediction throughput and efficiency.

For Beginners

Inference optimization makes your model’s predictions faster and more efficient.

Key features:

  • KV Cache: Remembers previous computations in attention layers (2-10x faster for long sequences)
  • Batching: Groups multiple predictions together (higher throughput)
  • Speculative Decoding: Uses a small model to draft tokens, then verifies (1.5-3x faster generation)

Default settings are optimized for most use cases. Simply enable and let the library handle the rest.

Example:

How It Works

This configuration controls advanced inference optimizations including KV caching for transformers, request batching for throughput, and speculative decoding for faster autoregressive generation. These optimizations are automatically applied during prediction based on your configuration.

Constructors

Constructor Summary
InferenceOptimizationConfig()

Properties

Property Summary
AttentionMaskingMode AttentionMasking { get; set; } Gets or sets how attention masking should be applied for optimized attention implementations.
CacheEvictionPolicy KVCacheEvictionPolicy { get; set; } Gets or sets the KV cache eviction policy.
InferenceOptimizationConfig Default { get; } Gets a default configuration with sensible settings for most use cases.
InferenceOptimizationConfig HighPerformance { get; } Gets a high-performance configuration optimized for maximum throughput.
InferenceQuantizationMode InferenceQuantization { get; set; } Gets or sets the weight quantization mode used for inference.
KVCachePrecisionMode KVCachePrecision { get; set; } Gets or sets the precision used for KV-cache storage.
KVCacheQuantizationMode KVCacheQuantization { get; set; } Gets or sets the quantization mode used for KV-cache storage.
PositionalEncodingType PositionalEncoding { get; set; } Gets or sets the positional encoding type for attention layers.
SpeculativeDecodingOptions SpeculativeDecoding { get; set; } Gets or sets the speculative-decoding tuning options (enable flag, speculation depth, policy, method, tree speculation).
bool AdaptiveBatchSize { get; set; } Gets or sets whether adaptive batch sizing is enabled.
bool EnableBatching { get; set; } Gets or sets whether request batching is enabled.
bool EnableFlashAttention { get; set; } Gets or sets whether Flash Attention is enabled (when applicable).
bool EnableKVCache { get; set; } Gets or sets whether KV (Key-Value) caching is enabled for attention layers.
bool EnableLayerFusion { get; set; } Gets or sets whether freeze-time layer fusion (BatchNorm folding) is applied when optimizing a model for inference.
bool EnablePagedKVCache { get; set; } Gets or sets whether to use a paged KV-cache backend (vLLM-style) for long-context / multi-sequence serving.
bool EnableWeightOnlyQuantization { get; set; } Gets or sets whether weight-only INT8 quantization is enabled for inference.
bool UseSlidingWindowKVCache { get; set; } Gets or sets whether to use a sliding window KV-cache for long contexts.
double RoPETheta { get; set; } Gets or sets the base frequency parameter for RoPE (theta).
int BatchTimeoutMs { get; set; } Gets or sets the maximum time to wait for batch to fill in milliseconds.
int KVCacheMaxSizeMB { get; set; } Gets or sets the maximum KV cache size in megabytes.
int KVCacheWindowSize { get; set; } Gets or sets the sliding window size in tokens when UseSlidingWindowKVCache is enabled.
int MaxBatchSize { get; set; } Gets or sets the maximum batch size for grouped predictions.
int MaxPrefillChunkTokens { get; set; } Gets or sets the maximum number of prompt tokens to prefill per scheduler step (chunked prefill).
int MinBatchSize { get; set; } Gets or sets the minimum batch size before processing.
int PagedKVCacheBlockSize { get; set; } Gets or sets the block size (in tokens) for the paged KV-cache when enabled.
int TensorParallelSize { get; set; } Gets or sets the number of tensor-parallel ranks (shards) the model is split across for serving.

Methods

Method Summary
InferenceOptimizationConfig Clone() Returns a shallow copy of this configuration.
void Validate() Validates the configuration and throws if any values are invalid.