DVoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
DVoRA (DoRA + VeRA) adapter - combines DoRA’s magnitude-direction decomposition with VeRA’s extreme parameter efficiency.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → DVoRAAdapter<T>
For Beginners
DVoRA is the ultimate parameter-efficient adapter.
Think of it as a hybrid technique:
- From DoRA: Separate magnitude (strength) from direction for stability
- From VeRA: Use shared random matrices and tiny scaling vectors for efficiency
- The magic: Apply VeRA’s adaptation only to the direction, not the magnitude
Parameter comparison for 1000x1000 layer with rank=8:
- Full fine-tuning: 1,000,000 parameters
- Standard LoRA: 16,000 parameters (98.4% reduction)
- DoRA: 17,000 parameters (LoRA + magnitude vector)
- VeRA: 1,600 parameters (99.84% reduction)
- DVoRA: ~1,600 parameters (same as VeRA!) but with better performance (5.0 vs 4.3)
Benefits:
- ✅ Extremely parameter-efficient (10x fewer than standard LoRA, same as VeRA)
- ✅ Better performance than VeRA alone (5.0 vs 4.3 score)
- ✅ Training stability from DoRA’s magnitude-direction decomposition
- ✅ Shared matrices reduce storage when adapting many layers
- ✅ Best choice for extreme memory constraints with quality requirements
Trade-offs:
- ⚠️ Requires shared matrix initialization before use
- ⚠️ Slightly more computation than VeRA (due to normalization)
- ⚠️ More complex than standard adapters (combines two techniques)
When to use DVoRA:
- Extreme memory constraints but need better quality than VeRA
- Mobile/edge deployment with limited resources
- Fine-tuning many layers efficiently
- When you want the absolute best parameter efficiency + quality balance
How It Works
DVoRA achieves the best of both worlds by:
- Applying DoRA’s magnitude-direction decomposition for training stability
- Using VeRA’s shared frozen matrices and scaling vectors for extreme parameter efficiency
- Applying the VeRA adaptation only to the direction component (not the magnitude)
Mathematical Formulation: Given pre-trained weights W, DVoRA:
- Decomposes: W = m * d (magnitude and direction)
- Adds the VeRA weight delta to the weights: V = W + ΔW, ΔW = d_scale * (B * A) * b_scale
- Takes the direction of the sum: d_norm = V / ||V|| (per output neuron)
- Recomposes: W’ = m * d_norm
Where:
- m: magnitude vector (trainable)
- d: direction matrix (normalized weight vectors)
- A, B: shared frozen random matrices (VeRA style)
- d_scale, b_scale: per-layer trainable scaling vectors (VeRA style)
Research Context: DVoRA scores 5.0 vs VeRA’s 4.3 (improvement of 16%) while maintaining ultra-low parameter counts. It combines DoRA’s superior training stability with VeRA’s extreme parameter efficiency.
References:
- DoRA: “Weight-Decomposed Low-Rank Adaptation” (ICML 2024 Oral)
- VeRA: “Vector-based Random Matrix Adaptation”
- DVoRA: Combines both techniques for optimal efficiency and performance
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new DVoRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured DVoRAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
DVoRAAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) |
Initializes a new DVoRA adapter wrapping an existing layer. |
Properties
| Property | Summary |
|---|---|
bool AreSharedMatricesInitialized { get; } |
Gets whether the shared matrices have been initialized. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the DVoRA adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void InitializeSharedMatrices(int inputSize, int outputSize, int rank, int? seed) |
Initializes the shared random matrices used by all DVoRA adapters. |
void ResetSharedMatrices() |
Resets the shared matrices (useful for testing or reinitializing). |
void ResetState() |
Resets the internal state of the DVoRA adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |