InferenceForwardContext

Models & Types · AiDotNet.Inference

Per-call inference context carried through the forward pass so that paged/cached attention layers can route to a specific KV-cache sequence at a specific position WITHOUT mutating shared layer state. This is what makes one optimized model instance + one shared PagedKVCache safe to drive for multiple concurrent generation sequences: the sequence id and position travel with the call instead of living in mutable layer fields.

How It Works

A single network forward over seqLen tokens uses ONE context: every context-aware layer reads Position as the start position for this forward and appends its KV at Position .. Position + seqLen. Layers do NOT advance Position (they all share the same start within a forward); the owning generation session advances it by the number of tokens after the whole forward completes.

Constructors

Constructor Summary
InferenceForwardContext(long sequenceId, int position) Creates a BATCHED context: batch row b maps to sequence sequenceIds[b] at positions[b].
InferenceForwardContext(long[] sequenceIds, int[] positions) Creates a BATCHED context: batch row b maps to sequence sequenceIds[b] at positions[b].

Properties

Property Summary
bool IsBatched { get; } True when this context addresses one sequence per batch row (batched decode).
int Position { get; set; } Start position (number of tokens already cached for this sequence) for this forward.
int[] Positions { get; } Per-row start positions, parallel to SequenceIds.
int[] RowLengths { get; set; } Per-row VALID token counts for a batched forward whose rows have different lengths (batched prefill: prompts differ in length, so a [batch, maxLen, dim] input is right-padded).
long SequenceId { get; } The KV-cache sequence id this forward belongs to (isolates concurrent sequences).
long[] SequenceIds { get; } When non-null, this forward is BATCHED: batch row b of a [batch, seqLen, dim] input belongs to sequence SequenceIds[b] at start position Positions[b].
string LoraTask { get; set; } When non-null, the LoRA task/adapter name this forward selects for MultiLoRAAdapter layers in single-sequence mode (null selects each adapter’s default active task).
string[] LoraTasks { get; set; } Per-row LoRA task names for a BATCHED forward, parallel to SequenceIds: entry b (null = default) selects the adapter for batch row b, so sequences requesting different adapters can co-batch in a single forward.