AiModelResultTransformerExtensions

Helpers & Utilities · AiDotNet.Transformers.Extensions

Model-specific inference extensions on AiModelResult for the transformer / language-model family. Part of #1836.

How It Works

See AiModelResultRadianceFieldExtensions for the full design rationale — extension methods live in the same assembly as AiModelResult so they access internal Model directly without exposing it.

Autoregressive generation is a common transformer inference pattern — the model outputs next-token logits per position, we sample/argmax the last position, append, and repeat. These extensions ship the two most common decoding strategies (greedy argmax + temperature sampling); more sophisticated strategies (beam search, top-k / top-p nucleus) can be layered on the same primitive.

Methods

Method Summary
Task<Tensor<T>> GenerateGreedyAsync(AiModelResult<T, TInput, TOutput> result, Tensor<T> startTokens, int maxNewTokens, int? eosTokenId, IProgress<int> onToken, CancellationToken cancellationToken) Async generation with per-token streaming callback.
Tensor<T> GenerateGreedy(AiModelResult<T, TInput, TOutput> result, Tensor<T> startTokens, int maxNewTokens, int? eosTokenId) Runs greedy autoregressive generation — at each step, pick the token with the highest logit.
Tensor<T> GenerateSampled(AiModelResult<T, TInput, TOutput> result, Tensor<T> startTokens, int maxNewTokens, T temperature, int? eosTokenId, int? seed) Runs temperature-sampled autoregressive generation.
Tensor<T>[] GenerateGreedyBatch(AiModelResult<T, TInput, TOutput> result, Tensor<T>[] startTokensPerPrompt, int maxNewTokens, int? eosTokenId) Batched greedy generation — runs N independent prompt sequences in parallel using the .NET thread pool.