Serving

21 public types in this namespace, organized by kind.

Models & Types (13)

Type Summary
BatchScheduler<T> Schedules sequences for continuous batching based on priority and resource constraints.
BatcherStatistics Statistics about the batcher’s operation.
GenerationRequest<T> Represents a request for text generation.
GenerationResult<T> Result of a generation request.
JsonGrammarConstraint An ITokenConstraint that restricts output to any well-formed JSON value of arbitrary nesting depth.
PositionLogProbs Log-probability information for one generated position: the chosen token’s log-probability plus, when requested, the top-N most likely alternatives at that step.
RegexTokenConstraint An ITokenConstraint that restricts generated text to match a regular expression.
SchedulerStatistics Statistics about the scheduler state.
SequenceCompletedEventArgs<T> Event args for sequence completion.
SequenceState<T> Represents the state of a single sequence being processed in continuous batching.
TokenFsmConstraint An ITokenConstraint backed by a compiled deterministic finite automaton (DFA) over token ids.
TokenGeneratedEventArgs<T> Event args for token generation.
TokenLogProb A single (token id, log-probability) pair, used to report the log-probabilities of candidate tokens at a decode position (OpenAI logprobs / top_logprobs).

Interfaces (1)

Type Summary
ITokenConstraint A per-sequence decoding constraint that restricts which vocabulary tokens may be emitted next, enabling structured / guided output (JSON, regex, grammar-constrained generation).

Enums (3)

Type Summary
SchedulingPolicy Scheduling policies for batch scheduling.
SequenceStatus Status of a sequence in the continuous batching system.
StopReason Reasons why generation stopped.

Options & Configuration (2)

Type Summary
BatchSchedulerConfig Configuration for the batch scheduler.
ContinuousBatcherConfig Configuration for the continuous batcher.

Helpers & Utilities (1)

Type Summary
JsonSchemaConstraint Compiles a (subset of) JSON Schema to a regular expression and builds a RegexTokenConstraint from it, so a model can be forced to emit JSON conforming to the schema.

Exceptions (1)

Type Summary
StructuredOutputConstraintException Thrown by an ITokenConstraint during masking when generation has reached a state from which the required format can no longer be satisfied — no valid non-EOS continuation remains and the text so far is not a complete valid instance (a non…