MixtureOfExpertsBuilder<T>

Layers · AiDotNet.NeuralNetworks.Layers

A builder class that helps create and configure Mixture-of-Experts layers with sensible defaults.

For Beginners

Think of this as a guided recipe for creating an MoE layer.

Instead of manually specifying every detail of your MoE layer (which experts to use, how to route between them, whether to use load balancing, etc.), this builder provides good default choices based on research and best practices.

It’s like having a cooking recipe that says “preheat to 350°F” instead of making you figure out the right temperature yourself. You can still customize if needed, but the defaults work well for most cases.

How It Works

This builder simplifies the creation of Mixture-of-Experts layers by providing convenient methods with research-backed default values. It follows best practices from MoE literature to ensure good initial configuration for most use cases.

Constructors

Constructor Summary
MixtureOfExpertsBuilder() Initializes a new instance of the MixtureOfExpertsBuilder class.

Methods

Method Summary
MixtureOfExpertsBuilder<T> WithDimensions(int inputDim, int outputDim) Sets the input and output dimensions for the MoE layer.
MixtureOfExpertsBuilder<T> WithExpertActivation(IActivationFunction<T> activation) Sets the activation function for experts.
MixtureOfExpertsBuilder<T> WithExpertHiddenDim(int hiddenDim) Sets the hidden dimension for the expert networks (for 2-layer experts).
MixtureOfExpertsBuilder<T> WithExperts(int numExperts) Sets the number of expert networks in the MoE layer.
MixtureOfExpertsBuilder<T> WithHiddenExpansion(int expansion) Sets the hidden dimension expansion factor for expert networks.
MixtureOfExpertsBuilder<T> WithIntermediateLayer(bool useIntermediateLayer) Configures whether experts should use an intermediate (hidden) layer.
MixtureOfExpertsBuilder<T> WithLoadBalancing(bool enabled, double weight) Configures load balancing to encourage even expert utilization.
MixtureOfExpertsBuilder<T> WithOutputActivation(IActivationFunction<T> activation) Sets the activation function for the MoE layer output.
MixtureOfExpertsBuilder<T> WithTopK(int k) Configures Top-K sparse routing.
MixtureOfExpertsLayer<T> Build() Builds the Mixture-of-Experts layer with the configured settings.