Inference
24 public types in this namespace, organized by kind.
Models & Types (15)
| Type |
Summary |
BlockManagerStats |
Statistics about the block manager state. |
BlockManager<T> |
Manages physical memory blocks for PagedAttention KV cache. |
BlockTable |
Maps logical block indices to physical block IDs for a sequence. |
BlockTableManager<T> |
Manages block tables for multiple sequences. |
CachedGroupedQueryAttention<T> |
Grouped-Query Attention with KV-Cache support for efficient autoregressive inference. |
CachedMultiHeadAttention<T> |
Multi-head attention layer with KV-Cache support for efficient autoregressive inference. |
DraftResult<T> |
Result of draft token generation returned by an IDraftModel: the guessed tokens and the probability distributions the draft assigned to them (used by the target model to accept/reject drafts). |
InferenceForwardContext |
Per-call inference context carried through the forward pass so that paged/cached attention layers can route to a specific KV-cache sequence at a specific position WITHOUT mutating shared layer state. |
KVCache<T> |
Key-Value cache for efficient autoregressive inference in transformer models. |
NGramDraftModel<T> |
A simple n-gram based draft model for speculative decoding. |
NeuralDraftModel<T> |
Ready-to-use IDraftModel that wraps any next-token-logits function as the draft model for speculative decoding. |
PagedAttentionKernel<T> |
Paged attention kernel that computes attention with block-based KV cache. |
PagedCachedMultiHeadAttention<T> |
Multi-head attention layer backed by PagedKVCache for efficient multi-sequence inference. |
PagedKVCacheStats |
Statistics about the paged KV cache. |
PagedKVCache<T> |
Paged key-value cache for efficient LLM serving memory management. |
Interfaces (1)
| Type |
Summary |
IDraftModel<T> |
Interface for draft models used in speculative decoding. |
Enums (2)
| Type |
Summary |
CacheDataType |
Data types supported for KV-Cache storage. |
CacheDevice |
Device placement options for KV-Cache. |
Structs (1)
Options & Configuration (4)
Helpers & Utilities (1)