Inference

24 public types in this namespace, organized by kind.

Models & Types (15)

Type Summary
BlockManagerStats Statistics about the block manager state.
BlockManager<T> Manages physical memory blocks for PagedAttention KV cache.
BlockTable Maps logical block indices to physical block IDs for a sequence.
BlockTableManager<T> Manages block tables for multiple sequences.
CachedGroupedQueryAttention<T> Grouped-Query Attention with KV-Cache support for efficient autoregressive inference.
CachedMultiHeadAttention<T> Multi-head attention layer with KV-Cache support for efficient autoregressive inference.
DraftResult<T> Result of draft token generation returned by an IDraftModel: the guessed tokens and the probability distributions the draft assigned to them (used by the target model to accept/reject drafts).
InferenceForwardContext Per-call inference context carried through the forward pass so that paged/cached attention layers can route to a specific KV-cache sequence at a specific position WITHOUT mutating shared layer state.
KVCache<T> Key-Value cache for efficient autoregressive inference in transformer models.
NGramDraftModel<T> A simple n-gram based draft model for speculative decoding.
NeuralDraftModel<T> Ready-to-use IDraftModel that wraps any next-token-logits function as the draft model for speculative decoding.
PagedAttentionKernel<T> Paged attention kernel that computes attention with block-based KV cache.
PagedCachedMultiHeadAttention<T> Multi-head attention layer backed by PagedKVCache for efficient multi-sequence inference.
PagedKVCacheStats Statistics about the paged KV cache.
PagedKVCache<T> Paged key-value cache for efficient LLM serving memory management.

Interfaces (1)

Type Summary
IDraftModel<T> Interface for draft models used in speculative decoding.

Enums (2)

Type Summary
CacheDataType Data types supported for KV-Cache storage.
CacheDevice Device placement options for KV-Cache.

Structs (1)

Type Summary
QuantizedWeights

Options & Configuration (4)

Type Summary
BlockManagerConfig Configuration for the block manager.
KVCacheConfig Configuration for Key-Value cache used in autoregressive inference.
PagedAttentionConfig Configuration for paged attention kernel.
PagedKVCacheConfig Configuration for PagedKVCache.

Helpers & Utilities (1)

Type Summary
Int8WeightOnlyQuantization