MoEModelBuilder<T>

Helpers & Utilities · AiDotNet.ModelLoading.Pretrained

Reconstructs a sparse mixture-of-experts decoder (Mixtral) from a Hugging Face HuggingFaceConfig and loads its pretrained weights. Same pre-LN RMSNorm + RoPE-GQA attention as the dense LLaMA builder, but the FFN is a MoEFeedForwardLayer (top-k-of-E gated experts). Produces a servable NeuralNetwork.

How It Works

Weight names follow Mixtral: block_sparse_moe.gate.weight is the router, and each expert has experts.{e}.w1 (gate), w3 (up), w2 (down). All projections are bias-free and stored [out, in], so they are transposed to the layer’s [in, out] on load. The MoE blocks are not recognized by the tensor-parallel partitioner, so imported MoE models serve through the base decode path.

Properties

Property Summary
IReadOnlyList<string> SupportedArchitectures { get; } The MoE architecture family names this builder recognizes.

Methods

Method Summary
NeuralNetwork<T> Build(HuggingFaceConfig config, INamedTensorSource weights, int warmupSequenceLength) Builds and weight-loads a Mixtral-style MoE decoder.