ITokenConstraint

Interfaces · AiDotNet.Serving.StructuredOutput

A per-sequence decoding constraint that restricts which vocabulary tokens may be emitted next, enabling structured / guided output (JSON, regex, grammar-constrained generation).

Known implementations: JsonGrammarConstraint, RegexTokenConstraint, TokenFsmConstraint

For Beginners

This is how “give me valid JSON” or “answer only yes/no” is enforced. At every step the model still produces scores for all tokens; the constraint blanks out the ones that would break the required format so the model is forced to stay on the rails, while still choosing freely among the tokens that keep the output valid.

How It Works

A constraint is a small state machine over generated tokens. Before each sampling step the engine calls ApplyMask to forbid tokens that would violate the target language; after a token is chosen it calls Accept to advance the state. The constraint is stateful and belongs to exactly one sequence — never share an instance across concurrent sequences.

Implementers: this interface is new in the serving layer and defines both IsComplete (the output so far is a complete valid instance) and IsTerminal (complete AND no valid non-EOS continuation remains — where the engine stops). A custom constraint must implement both; a common pattern is IsTerminal => IsComplete && !HasNonEosContinuation. The built-in constraints (JSON grammar/schema, regex, token FSM) are the reference implementations.

Properties

Property Summary
bool IsComplete { get; } Gets a value indicating whether the generated text so far is accepting — a complete valid instance of the target language.
bool IsTerminal { get; } Gets a value indicating whether the constraint is terminal: the generated text so far is a complete valid instance of the target language AND no valid non-EOS continuation remains, so the engine may stop the sequence.

Methods

Method Summary
void Accept(int tokenId) Advances the constraint state after tokenId has been committed to the sequence.
void ApplyMask(Span<float> logits) Masks tokens that are not permitted as the next token in-place: sets logits[i] = for every token id i that would violate the constraint given the tokens accepted so far.