XLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
X-LoRA (Mixture of LoRA Experts) adapter that uses multiple LoRA experts with learned routing.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → XLoRAAdapter<T>
For Beginners
X-LoRA is like having multiple specialists instead of one generalist.
Think of it like this:
- Standard LoRA: One adapter tries to handle all tasks
- X-LoRA: Multiple expert adapters, each specializing in different patterns
- A “gating network” decides which experts to use for each input
Real-world analogy: Instead of one doctor handling all patients, you have:
- Expert 1: Specializes in one type of pattern (e.g., cat images)
- Expert 2: Specializes in another pattern (e.g., dog images)
- Expert 3: Handles other cases
- Gating network: Looks at each input and decides which expert(s) to consult
Benefits:
- More capacity: Multiple experts can learn different aspects
- Better specialization: Each expert focuses on what it’s good at
- Dynamic routing: Different inputs activate different experts
- Efficient: Only computes what’s needed for each input
Example: For a 1000x1000 layer with 4 experts at rank=4 each:
- Total LoRA parameters: 4 * (4 * 1000 + 4 * 1000) = 32,000 parameters
- Gating network: ~1000 parameters
- Total: ~33,000 parameters (still 96.7% reduction from 1M!)
- But with more capacity than single rank=16 LoRA (32,000 params)
Trade-offs:
- More flexible: Experts specialize in different patterns
- Better performance: Often outperforms single LoRA at same parameter count
- Dynamic routing: Adapts to different inputs
- More complex: Requires training gating network
- Slightly slower: Must compute multiple experts and gating weights
Reference: “Mixture of LoRA Experts” (X-LoRA)
https://arxiv.org/abs/2402.07148
How It Works
X-LoRA extends standard LoRA by using a mixture of experts approach:
- Multiple LoRA adapters (“experts”) are applied to the same layer
- A gating network learns to weight each expert’s contribution based on the input
- Different inputs may activate different experts, allowing for more flexible adaptation
- This provides greater capacity than a single LoRA adapter with the same total rank
The forward pass computes:
- base_output = base_layer(input)
- For each expert i: expert_output[i] = lora_expert[i](input)
- gating_weights = softmax(gating_network(input))
- final_lora_output = sum(gating_weights[i] * expert_output[i])
- output = base_output + final_lora_output
Constructors
| Constructor | Summary |
|---|---|
XLoRAAdapter(ILayer<T> baseLayer, int numberOfExperts, int expertRank, double alpha, bool freezeBaseLayer) |
Initializes a new X-LoRA adapter with the specified parameters. |
Properties
| Property | Summary |
|---|---|
DenseLayer<T> GatingNetwork { get; } |
Gets the gating network used for routing. |
LoRALayer<T>[] Experts { get; } |
Gets the array of LoRA expert layers. |
int NumberOfExpertss { get; } |
Gets the number of LoRA experts in this adapter. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges all LoRA expert adaptations into the base layer and returns the merged layer. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
Tensor<T> GetLastGatingWeights() |
Gets the gating weights from the last forward pass. |
void ResetState() |
Resets the internal state of the base layer, all experts, and the gating network. |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |