Serving
21 public types in this namespace, organized by kind.
Models & Types (13)
| Type |
Summary |
BatchScheduler<T> |
Schedules sequences for continuous batching based on priority and resource constraints. |
BatcherStatistics |
Statistics about the batcher’s operation. |
GenerationRequest<T> |
Represents a request for text generation. |
GenerationResult<T> |
Result of a generation request. |
JsonGrammarConstraint |
An ITokenConstraint that restricts output to any well-formed JSON value of arbitrary nesting depth. |
PositionLogProbs |
Log-probability information for one generated position: the chosen token’s log-probability plus, when requested, the top-N most likely alternatives at that step. |
RegexTokenConstraint |
An ITokenConstraint that restricts generated text to match a regular expression. |
SchedulerStatistics |
Statistics about the scheduler state. |
SequenceCompletedEventArgs<T> |
Event args for sequence completion. |
SequenceState<T> |
Represents the state of a single sequence being processed in continuous batching. |
TokenFsmConstraint |
An ITokenConstraint backed by a compiled deterministic finite automaton (DFA) over token ids. |
TokenGeneratedEventArgs<T> |
Event args for token generation. |
TokenLogProb |
A single (token id, log-probability) pair, used to report the log-probabilities of candidate tokens at a decode position (OpenAI logprobs / top_logprobs). |
Interfaces (1)
| Type |
Summary |
ITokenConstraint |
A per-sequence decoding constraint that restricts which vocabulary tokens may be emitted next, enabling structured / guided output (JSON, regex, grammar-constrained generation). |
Enums (3)
Options & Configuration (2)
Helpers & Utilities (1)
| Type |
Summary |
JsonSchemaConstraint |
Compiles a (subset of) JSON Schema to a regular expression and builds a RegexTokenConstraint from it, so a model can be forced to emit JSON conforming to the schema. |
Exceptions (1)
| Type |
Summary |
StructuredOutputConstraintException |
Thrown by an ITokenConstraint during masking when generation has reached a state from which the required format can no longer be satisfied — no valid non-EOS continuation remains and the text so far is not a complete valid instance (a non… |