LoRAPlusAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

LoRA+ adapter that uses optimized learning rates for faster convergence and better performance.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → LoRAPlusAdapter<T>

For Beginners

LoRA+ is an enhanced version of LoRA that trains faster and better.

In standard LoRA:

  • Both matrix A and B are updated with the same learning rate
  • Matrix B starts at zero, so it needs time to “catch up”
  • Matrix A starts random, so it’s already contributing from the start

LoRA+ recognizes this asymmetry:

  • Matrix A is updated with a base learning rate (e.g., 0.0001)
  • Matrix B is updated with a higher learning rate (e.g., 0.0016 = 16x higher)
  • This accelerates learning without instability

Key parameters:

  • BaseLearningRate: Learning rate for matrix A (the “slow” matrix)
  • LearningRateRatio: Multiplier for matrix B (typically 16.0)
  • ScaledLearningRate: Computed as BaseLearningRate * LearningRateRatio

Research shows LoRA+ typically achieves:

  • 2x faster convergence
  • Better final performance
  • No additional parameters compared to standard LoRA

Example: If base learning rate is 0.0001 and ratio is 16.0:

  • Matrix A updates with learning rate 0.0001
  • Matrix B updates with learning rate 0.0016

Reference: LoRA+: Efficient Low Rank Adaptation of Large Models (February 2024)

How It Works

LoRA+ (February 2024) improves upon standard LoRA by using different learning rates for the A and B matrices. The key insight is that matrix B (which starts at zero) needs faster updates than matrix A (which starts random). This simple modification leads to significantly faster convergence and improved final performance.

Example

using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;

var adapter = new LoRAPlusAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured LoRAPlusAdapter (rank {config.Rank}).");

Constructors

Constructor Summary
LoRAPlusAdapter(ILayer<T> baseLayer, int rank, double alpha, double learningRateRatio, bool freezeBaseLayer) Initializes a new LoRA+ adapter with optimized dual learning rates.

Properties

Property Summary
T BaseLearningRate { get; } Gets the base learning rate for matrix A.
T ScaledLearningRate { get; } Gets the scaled learning rate for matrix B.
double LearningRateRatio { get; set; } Gets or sets the learning rate ratio between matrix B and matrix A.

Methods

Method Summary
ILayer<T> MergeToOriginalLayer() Merges the LoRA+ adaptation into the base layer and returns the merged layer.
void SetLearningRates(T baseLearningRate) Sets the learning rates for this adapter.
void UpdateParameters(T learningRate) Updates parameters using dual learning rates (base rate for A, scaled rate for B).