AdaLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Adaptive Low-Rank Adaptation (AdaLoRA) adapter that dynamically allocates parameter budgets among weight matrices.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → AdaLoRAAdapter<T>
For Beginners
AdaLoRA is like smart LoRA that learns which parts of the adaptation matter most.
Think of standard LoRA as giving every layer the same budget (rank=8 everywhere). AdaLoRA is smarter:
- Some layers get more budget (rank=16) because they’re important for the task
- Other layers get less budget (rank=2) because small changes are enough
- The model learns this automatically during training
How it works:
- Start with a large rank (e.g., maxRank=32)
- During training, track how important each component is
- Prune components with low importance scores
- Focus parameters on what actually helps
Benefits:
- More parameter-efficient than fixed-rank LoRA
- Better performance with same parameter budget
- Automatically finds optimal rank per layer
Reference: “Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning” (ICLR 2023)
https://arxiv.org/abs/2303.10512
How It Works
AdaLoRA improves upon standard LoRA by dynamically adjusting the rank allocation based on importance scores. Instead of using a fixed rank for all weight matrices, AdaLoRA:
- Starts with a maximum rank and adaptively reduces it during training
- Computes importance scores for each singular value component
- Prunes less important components to focus parameter budget on critical adaptations
- Allows different layers to have different effective ranks
This leads to more efficient parameter usage compared to fixed-rank LoRA, especially for large models where some layers need more adaptation capacity than others.
Constructors
| Constructor | Summary |
|---|---|
AdaLoRAAdapter(ILayer<T> baseLayer, int maxRank, double alpha, bool freezeBaseLayer, double rankPruningThreshold, int minRank, int pruningInterval, double importanceScoreEMA) |
Initializes a new AdaLoRA adapter with adaptive rank allocation. |
Properties
| Property | Summary |
|---|---|
int CurrentRank { get; } |
Gets the current active rank after pruning. |
int MaxRank { get; } |
Gets the maximum rank this adapter can use. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeToOriginalLayer() |
Merges the AdaLoRA adaptation into the base layer and returns the merged layer. |
Vector<T> GetImportanceScores() |
Gets a copy of the current importance scores. |
void ExpandRank(int additionalRank) |
Expands the rank by adding new components (for cases where more capacity is needed). |