InferenceForwardContext
Models & Types · AiDotNet.Inference
Per-call inference context carried through the forward pass so that paged/cached attention
layers can route to a specific KV-cache sequence at a specific position WITHOUT mutating shared
layer state. This is what makes one optimized model instance + one shared PagedKVCache
safe to drive for multiple concurrent generation sequences: the sequence id and position travel
with the call instead of living in mutable layer fields.
How It Works
A single network forward over seqLen tokens uses ONE context: every context-aware layer
reads Position as the start position for this forward and appends its KV at
Position .. Position + seqLen. Layers do NOT advance Position (they all
share the same start within a forward); the owning generation session advances it by the number
of tokens after the whole forward completes.
Constructors
| Constructor | Summary |
|---|---|
InferenceForwardContext(long sequenceId, int position) |
Creates a BATCHED context: batch row b maps to sequence sequenceIds[b] at positions[b]. |
InferenceForwardContext(long[] sequenceIds, int[] positions) |
Creates a BATCHED context: batch row b maps to sequence sequenceIds[b] at positions[b]. |
Properties
| Property | Summary |
|---|---|
bool IsBatched { get; } |
True when this context addresses one sequence per batch row (batched decode). |
int Position { get; set; } |
Start position (number of tokens already cached for this sequence) for this forward. |
int[] Positions { get; } |
Per-row start positions, parallel to SequenceIds. |
int[] RowLengths { get; set; } |
Per-row VALID token counts for a batched forward whose rows have different lengths (batched prefill: prompts differ in length, so a [batch, maxLen, dim] input is right-padded). |
long SequenceId { get; } |
The KV-cache sequence id this forward belongs to (isolates concurrent sequences). |
long[] SequenceIds { get; } |
When non-null, this forward is BATCHED: batch row b of a [batch, seqLen, dim] input belongs to sequence SequenceIds[b] at start position Positions[b]. |
string LoraTask { get; set; } |
When non-null, the LoRA task/adapter name this forward selects for MultiLoRAAdapter layers in single-sequence mode (null selects each adapter’s default active task). |
string[] LoraTasks { get; set; } |
Per-row LoRA task names for a BATCHED forward, parallel to SequenceIds: entry b (null = default) selects the adapter for batch row b, so sequences requesting different adapters can co-batch in a single forward. |