AdaLoRAAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

Adaptive Low-Rank Adaptation (AdaLoRA) adapter that dynamically allocates parameter budgets among weight matrices.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → AdaLoRAAdapter<T>

For Beginners

AdaLoRA is like smart LoRA that learns which parts of the adaptation matter most.

Think of standard LoRA as giving every layer the same budget (rank=8 everywhere). AdaLoRA is smarter:

  • Some layers get more budget (rank=16) because they’re important for the task
  • Other layers get less budget (rank=2) because small changes are enough
  • The model learns this automatically during training

How it works:

  1. Start with a large rank (e.g., maxRank=32)
  2. During training, track how important each component is
  3. Prune components with low importance scores
  4. Focus parameters on what actually helps

Benefits:

  • More parameter-efficient than fixed-rank LoRA
  • Better performance with same parameter budget
  • Automatically finds optimal rank per layer

Reference: “Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning” (ICLR 2023) https://arxiv.org/abs/2303.10512

How It Works

AdaLoRA improves upon standard LoRA by dynamically adjusting the rank allocation based on importance scores. Instead of using a fixed rank for all weight matrices, AdaLoRA:

  • Starts with a maximum rank and adaptively reduces it during training
  • Computes importance scores for each singular value component
  • Prunes less important components to focus parameter budget on critical adaptations
  • Allows different layers to have different effective ranks

This leads to more efficient parameter usage compared to fixed-rank LoRA, especially for large models where some layers need more adaptation capacity than others.

Constructors

Constructor Summary
AdaLoRAAdapter(ILayer<T> baseLayer, int maxRank, double alpha, bool freezeBaseLayer, double rankPruningThreshold, int minRank, int pruningInterval, double importanceScoreEMA) Initializes a new AdaLoRA adapter with adaptive rank allocation.

Properties

Property Summary
int CurrentRank { get; } Gets the current active rank after pruning.
int MaxRank { get; } Gets the maximum rank this adapter can use.

Methods

Method Summary
ILayer<T> MergeToOriginalLayer() Merges the AdaLoRA adaptation into the base layer and returns the merged layer.
Vector<T> GetImportanceScores() Gets a copy of the current importance scores.
void ExpandRank(int additionalRank) Expands the rank by adding new components (for cases where more capacity is needed).