SequenceState<T>

Models & Types · AiDotNet.Serving.ContinuousBatching

Represents the state of a single sequence being processed in continuous batching.

For Beginners

Think of this as tracking one person’s order in a restaurant.

Traditional batching: Everyone orders at once, waits together, gets food together. Continuous batching: People can order anytime, food comes when ready, new orders join ongoing batch.

SequenceState tracks:

  • What tokens have been generated so far
  • When this request started
  • Whether generation is complete
  • How many tokens are left to generate

How It Works

Each sequence tracks its own progress through generation, including tokens generated, KV-cache state, and completion status. This enables sequences to be added to and removed from batches dynamically.

Constructors

Constructor Summary
SequenceState(GenerationRequest<T> request) Creates a new sequence state from a generation request.

Properties

Property Summary
DateTime CreatedAt { get; } Timestamp when the sequence was created.
DateTime? CompletedAt { get; set; } Timestamp when generation completed.
DateTime? GenerationStartedAt { get; set; } Timestamp when generation started (after prefill).
GenerationRequest<T> Request { get; } The original request that created this sequence.
List<PositionLogProbs> LogProbs { get; set; } Per-generated-token log-probability records, populated only when the request set IncludeLogProbs.
List<int> TokenIds { get; } List of token IDs generated so far (including prompt tokens).
SequenceStatus Status { get; set; } Current status of this sequence.
StopReason? FinishReason { get; set; } Stop reason if generation is complete.
TimeSpan QueueTime { get; } Gets the time spent in queue (before generation started).
TimeSpan? GenerationTime { get; } Gets the total generation time (after prefill).
bool PrefillComplete { get; set; } Whether the prefill phase is complete.
double CumulativeLogProb { get; set; } Cumulative log probability of generated tokens.
double? TokensPerSecond { get; } Gets tokens per second for this sequence.
int BatchIndex { get; set; } Index in the current batch (-1 if not in batch).
int CacheSlot { get; set; } Cache slot index for this sequence.
int GeneratedLength { get; } Number of tokens generated (excluding prompt).
int MaxNewTokens { get; } Maximum number of new tokens to generate.
int PrefillCursor { get; set; } Number of prompt tokens whose KV has been prefilled so far.
int Priority { get; set; } Priority for scheduling (higher = more important).
int PromptLength { get; } Number of tokens from the original prompt.
long SequenceId { get; } Unique identifier for this sequence.
object UserContext { get; set; } Optional user context associated with this sequence.
string ErrorMessage { get; set; } An optional human-readable error message, set when the sequence failed (see Fail).

Methods

Method Summary
bool ShouldStop(int eosTokenId, IReadOnlyCollection<int> stopTokenIds) Checks if generation should stop based on various conditions.
void AppendToken(int tokenId, double logProb) Appends a newly generated token to the sequence.
void Cancel() Marks the sequence as cancelled.
void Complete(StopReason reason) Marks the sequence as complete.
void Fail(string errorMessage) Marks the sequence as failed.