DyLoRAAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

DyLoRA (Dynamic LoRA) adapter that trains with multiple ranks simultaneously.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → DyLoRAAdapter<T>

For Beginners

DyLoRA is like LoRA with a superpower - flexibility!

Standard LoRA problem:

  • You choose rank=8 and train
  • Later realize rank=4 would work fine (save memory/speed)
  • Or need rank=16 for better quality
  • Must retrain from scratch with the new rank

DyLoRA solution:

  • Train once with multiple ranks (e.g., [2, 4, 8, 16])
  • Deploy with ANY of those ranks without retraining
  • Switch between ranks at runtime based on device capabilities

How it works:

  1. Train with MaxRank (e.g., 16) but randomly use smaller ranks during training
  2. Nested dropout ensures each rank works independently
  3. After training, pick deployment rank based on needs (2=fastest, 16=best quality)

Use cases:

  • Deploy same model to mobile (rank=2) and server (rank=16)
  • Dynamic quality scaling based on battery level
  • A/B testing different rank/quality trade-offs
  • Training once, deploying everywhere

Example: Train with ActiveRanks=[2,4,8], deploy with:

  • Rank=2 for mobile devices (98% parameter reduction, good quality)
  • Rank=4 for tablets (95% parameter reduction, better quality)
  • Rank=8 for desktops (90% parameter reduction, best quality)

How It Works

DyLoRA extends the standard LoRA approach by training multiple rank configurations simultaneously using a nested dropout technique. This allows a single trained adapter to be deployed at different rank levels without retraining, providing flexibility for different hardware constraints or performance requirements.

The key innovation is nested dropout: during training, for each forward pass, a random rank r is selected from the active ranks, and only the first r components of matrices A and B are used. This ensures that smaller ranks can function independently and don’t rely on higher-rank components.

Constructors

Constructor Summary
DyLoRAAdapter(ILayer<T> baseLayer, int maxRank, int[] activeRanks, double alpha, bool freezeBaseLayer) Initializes a new DyLoRA adapter with the specified parameters.

Properties

Property Summary
bool IsTraining { get; set; } Gets or sets whether the adapter is in training mode.
int CurrentDeploymentRank { get; set; } Gets or sets the current deployment rank used during inference.
int MaxRank { get; } Gets the maximum rank of the DyLoRA adapter.
int[] ActiveRanks { get; } Gets the array of active ranks used during training.

Methods

Method Summary
ILayer<T> MergeToOriginalLayer() Merges the DyLoRA adaptation into the base layer using the current deployment rank.
void Eval() Sets the adapter to evaluation mode (uses fixed deployment rank).
void SetDeploymentRank(int rank) Sets the deployment rank for inference.
void Train() Sets the adapter to training mode (enables nested dropout).
void TrainWithNestedDropout(Tensor<T>[] inputs, Tensor<T>[] targets, int epochs, T learningRate, Func<Tensor<T>, Tensor<T>, T> lossFunction) Trains the adapter with nested dropout across all active ranks, differentiating a caller-supplied tensor-valued loss.
void TrainWithNestedDropout(Tensor<T>[] inputs, Tensor<T>[] targets, int epochs, T learningRate, Func<Tensor<T>, Tensor<T>, Tensor<T>> lossObjective) Trains the adapter with nested dropout across all active ranks, differentiating a caller-supplied tensor-valued loss.
void UpdateParameters(T learningRate) Updates parameters for the base layer and the LoRA layer using cached gradients.