DecoderOptions<T>

Options & Configuration · AiDotNet.ModelLoading.Pretrained

Per-family knobs for LlamaModelBuilder that capture the small, faithful differences between LLaMA-family decoders (LLaMA/Mistral/Qwen2 use the defaults; Gemma flips the norm and embedding options; Phi-3 pre-splits its fused projections before build so it also uses the defaults).

Constructors

Constructor Summary
DecoderOptions()

Properties

Property Summary
DecoderOptions<T> Gemma { get; } The Gemma decoder options: GeGLU (tanh GELU), (1 + weight) RMSNorm, √hidden embedding scale.
DecoderOptions<T> Llama { get; } The LLaMA/Mistral defaults (SiLU gate, plain RMSNorm, no embedding scaling, no attn bias).
DecoderOptions<T> Qwen2 { get; } Qwen2 options: LLaMA-style but with q/k/v attention projection biases.
IActivationFunction<T> FfnActivation { get; set; } The gated-FFN activation applied on the gate path.
bool RmsNormAddsOne { get; set; } When true, RMSNorm scales by (1 + weight) rather than weight — the Gemma convention.
bool ScaleEmbeddingBySqrtHidden { get; set; } When true, the token embeddings are multiplied by sqrt(hidden_size) — the Gemma embedding normalizer.
bool UseAttentionQkvBias { get; set; } When true, the attention q/k/v projections carry biases (Qwen2); the output projection’s bias is loaded when present and left zero otherwise.