PagedAttentionKernel<T>

Models & Types · AiDotNet.Inference.PagedAttention

Paged attention kernel that computes attention with block-based KV cache.

For Beginners

Normal attention reads KV cache like reading a book page by page.

Paged attention is like reading a book where pages are scattered:

  1. Look up where each page is stored (block table)
  2. Go to that location (physical block)
  3. Read the content (KV values)
  4. Continue with next page

The extra lookups add slight overhead, but the memory savings are huge!

How It Works

This kernel performs attention computation using the paged KV cache structure. Instead of accessing KV tensors contiguously, it uses block tables to find the physical locations of each token’s KV data.

Constructors

Constructor Summary
PagedAttentionKernel(PagedKVCache<T> kvCache, PagedAttentionConfig config) Creates a new paged attention kernel.

Properties

Property Summary
PagedAttentionConfig Config { get; } Gets the configuration.

Methods

Method Summary
void ComputeAttention(ReadOnlySpan<float> query, long sequenceId, int layer, Span<float> output, float scale, bool causalMask) Computes paged attention for a single query token.
void ComputeBatchedAttention(ReadOnlySpan<float> queries, long[] sequenceIds, int layer, Span<float> outputs, float scale) Computes paged attention for a batch of queries.
void ComputeContiguousCausalPrefill(ReadOnlySpan<float> queries, ReadOnlySpan<float> keys, ReadOnlySpan<float> values, int rowLen, Span<float> output, float scale, float[] alibiSlopes) Computes causal self-attention for a fresh prefill block whose Q/K/V are supplied directly as contiguous, already-projected (and, for RoPE models, already-rotated) buffers laid out [rowLen, numHeads*headDim] in position order.
void ComputeTiledPagedAttention(ReadOnlySpan<float> query, long sequenceId, int layer, Span<float> output, float scale, float[] alibiSlopes, int queryPosition) Computes paged attention with Flash Attention-style tiling.
void Forward(ReadOnlySpan<float> hiddenStates, ReadOnlySpan<float> wQ, ReadOnlySpan<float> wK, ReadOnlySpan<float> wV, ReadOnlySpan<float> wO, long sequenceId, int position, int layer, Span<float> output) Performs a full forward pass: projects QKV, updates cache, computes attention.
void ForwardQuantized(ReadOnlySpan<float> hiddenStates, QuantizedWeights wQ, QuantizedWeights wK, QuantizedWeights wV, QuantizedWeights wO, long sequenceId, int position, int layer, Span<float> output)
void UpdateCache(ReadOnlySpan<float> key, ReadOnlySpan<float> value, long sequenceId, int position, int layer) Updates the KV cache with new key and value tensors.