GenerationRequest<T>

Models & Types · AiDotNet.Serving.ContinuousBatching

Represents a request for text generation.

Constructors

Constructor Summary
GenerationRequest()

Properties

Property Summary
Action<int> OnTokenGenerated { get; set; } Callback for streaming tokens.
IReadOnlyDictionary<int, float> LogitBias { get; set; } Optional per-token additive logit bias (OpenAI logit_bias): token id -> bias added to the token’s logit before sampling.
ITokenConstraint Constraint { get; set; } Optional structured-output constraint (JSON, regex, grammar, or choice) that restricts which tokens may be emitted so the output conforms to a required format.
List<int> PromptTokenIds { get; set; } Token IDs of the prompt.
List<int> StopTokenIds { get; set; } Additional stop token IDs.
bool IncludeLogProbs { get; set; } Whether to record per-token log-probabilities during generation (OpenAI logprobs).
bool UseBeamSearch { get; set; } Whether to use beam search.
float FrequencyPenalty { get; set; } OpenAI frequency_penalty (default 0): each token’s logit is reduced by this value times the number of times it has already appeared in the text so far, discouraging verbatim repetition.
float MinP { get; set; } Min-p sampling threshold: drop tokens below this fraction of the top token’s probability (0 = disabled).
float PresencePenalty { get; set; } OpenAI presence_penalty (default 0): a token’s logit is reduced by this value once if it has appeared at all in the text so far, encouraging the model to introduce new tokens/topics.
float RepetitionPenalty { get; set; } Repetition penalty (1.0 = no penalty).
float Temperature { get; set; } Temperature for sampling (higher = more random).
float TopP { get; set; } Top-p (nucleus) sampling threshold.
int MaxNewTokens { get; set; } Maximum number of new tokens to generate.
int NumBeams { get; set; } Number of beams for beam search.
int Priority { get; set; } Priority for scheduling (higher = more important).
int TopK { get; set; } Top-k sampling (0 = disabled).
int TopLogProbs { get; set; } How many of the most likely alternative tokens to record per position (OpenAI top_logprobs, 0-20).
int? EosTokenId { get; set; } Optional end-of-sequence token id for this request.
int? Seed { get; set; } Optional RNG seed for reproducible sampling.
int? SpeculationDepth { get; set; } Optional per-request speculative-decoding draft depth.
object UserContext { get; set; } Optional user context.
string AdapterId { get; set; } Optional multi-LoRA adapter name to activate for this request (S-LoRA-style serving).