SequenceState<T>
Models & Types · AiDotNet.Serving.ContinuousBatching
Represents the state of a single sequence being processed in continuous batching.
For Beginners
Think of this as tracking one person’s order in a restaurant.
Traditional batching: Everyone orders at once, waits together, gets food together. Continuous batching: People can order anytime, food comes when ready, new orders join ongoing batch.
SequenceState tracks:
- What tokens have been generated so far
- When this request started
- Whether generation is complete
- How many tokens are left to generate
How It Works
Each sequence tracks its own progress through generation, including tokens generated, KV-cache state, and completion status. This enables sequences to be added to and removed from batches dynamically.
Constructors
| Constructor | Summary |
|---|---|
SequenceState(GenerationRequest<T> request) |
Creates a new sequence state from a generation request. |
Properties
| Property | Summary |
|---|---|
DateTime CreatedAt { get; } |
Timestamp when the sequence was created. |
DateTime? CompletedAt { get; set; } |
Timestamp when generation completed. |
DateTime? GenerationStartedAt { get; set; } |
Timestamp when generation started (after prefill). |
GenerationRequest<T> Request { get; } |
The original request that created this sequence. |
List<PositionLogProbs> LogProbs { get; set; } |
Per-generated-token log-probability records, populated only when the request set IncludeLogProbs. |
List<int> TokenIds { get; } |
List of token IDs generated so far (including prompt tokens). |
SequenceStatus Status { get; set; } |
Current status of this sequence. |
StopReason? FinishReason { get; set; } |
Stop reason if generation is complete. |
TimeSpan QueueTime { get; } |
Gets the time spent in queue (before generation started). |
TimeSpan? GenerationTime { get; } |
Gets the total generation time (after prefill). |
bool PrefillComplete { get; set; } |
Whether the prefill phase is complete. |
double CumulativeLogProb { get; set; } |
Cumulative log probability of generated tokens. |
double? TokensPerSecond { get; } |
Gets tokens per second for this sequence. |
int BatchIndex { get; set; } |
Index in the current batch (-1 if not in batch). |
int CacheSlot { get; set; } |
Cache slot index for this sequence. |
int GeneratedLength { get; } |
Number of tokens generated (excluding prompt). |
int MaxNewTokens { get; } |
Maximum number of new tokens to generate. |
int PrefillCursor { get; set; } |
Number of prompt tokens whose KV has been prefilled so far. |
int Priority { get; set; } |
Priority for scheduling (higher = more important). |
int PromptLength { get; } |
Number of tokens from the original prompt. |
long SequenceId { get; } |
Unique identifier for this sequence. |
object UserContext { get; set; } |
Optional user context associated with this sequence. |
string ErrorMessage { get; set; } |
An optional human-readable error message, set when the sequence failed (see Fail). |
Methods
| Method | Summary |
|---|---|
bool ShouldStop(int eosTokenId, IReadOnlyCollection<int> stopTokenIds) |
Checks if generation should stop based on various conditions. |
void AppendToken(int tokenId, double logProb) |
Appends a newly generated token to the sequence. |
void Cancel() |
Marks the sequence as cancelled. |
void Complete(StopReason reason) |
Marks the sequence as complete. |
void Fail(string errorMessage) |
Marks the sequence as failed. |