DecoderOptions<T>
Options & Configuration · AiDotNet.ModelLoading.Pretrained
Per-family knobs for LlamaModelBuilder that capture the small, faithful differences
between LLaMA-family decoders (LLaMA/Mistral/Qwen2 use the defaults; Gemma flips the norm and embedding
options; Phi-3 pre-splits its fused projections before build so it also uses the defaults).
Constructors
| Constructor | Summary |
|---|---|
DecoderOptions() |
Properties
| Property | Summary |
|---|---|
DecoderOptions<T> Gemma { get; } |
The Gemma decoder options: GeGLU (tanh GELU), (1 + weight) RMSNorm, √hidden embedding scale. |
DecoderOptions<T> Llama { get; } |
The LLaMA/Mistral defaults (SiLU gate, plain RMSNorm, no embedding scaling, no attn bias). |
DecoderOptions<T> Qwen2 { get; } |
Qwen2 options: LLaMA-style but with q/k/v attention projection biases. |
IActivationFunction<T> FfnActivation { get; set; } |
The gated-FFN activation applied on the gate path. |
bool RmsNormAddsOne { get; set; } |
When true, RMSNorm scales by (1 + weight) rather than weight — the Gemma convention. |
bool ScaleEmbeddingBySqrtHidden { get; set; } |
When true, the token embeddings are multiplied by sqrt(hidden_size) — the Gemma embedding normalizer. |
bool UseAttentionQkvBias { get; set; } |
When true, the attention q/k/v projections carry biases (Qwen2); the output projection’s bias is loaded when present and left zero otherwise. |