MoEModelBuilder<T>
Helpers & Utilities · AiDotNet.ModelLoading.Pretrained
Reconstructs a sparse mixture-of-experts decoder (Mixtral) from a Hugging Face
HuggingFaceConfig and loads its pretrained weights. Same pre-LN RMSNorm + RoPE-GQA
attention as the dense LLaMA builder, but the FFN is a MoEFeedForwardLayer
(top-k-of-E gated experts). Produces a servable NeuralNetwork.
How It Works
Weight names follow Mixtral: block_sparse_moe.gate.weight is the router, and each expert has
experts.{e}.w1 (gate), w3 (up), w2 (down). All projections are bias-free and stored
[out, in], so they are transposed to the layer’s [in, out] on load. The MoE blocks are not
recognized by the tensor-parallel partitioner, so imported MoE models serve through the base decode path.
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedArchitectures { get; } |
The MoE architecture family names this builder recognizes. |
Methods
| Method | Summary |
|---|---|
NeuralNetwork<T> Build(HuggingFaceConfig config, INamedTensorSource weights, int warmupSequenceLength) |
Builds and weight-loads a Mixtral-style MoE decoder. |