VeRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
VeRA (Vector-based Random Matrix Adaptation) adapter - an extreme parameter-efficient variant of LoRA.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → VeRAAdapter<T>
For Beginners
VeRA is an ultra-efficient version of LoRA for extreme memory constraints.
Think of the difference this way:
- Standard LoRA: Each layer has its own pair of small matrices (A and B) that are trained
- VeRA: ALL layers share the same random matrices (A and B) which are frozen. Only tiny
scaling vectors are trained per layer.
Example parameter comparison for a 1000x1000 layer with rank=8:
- Full fine-tuning: 1,000,000 parameters
- Standard LoRA (rank=8): 16,000 parameters (98.4% reduction)
- VeRA (rank=8): ~1,600 parameters (99.84% reduction) - 10x fewer than LoRA!
Trade-offs:
- ✅ Extreme parameter efficiency (10x fewer than LoRA)
- ✅ Very low memory footprint
- ✅ Shared matrices reduce storage when adapting many layers
- ⚠️ Slightly less flexible than standard LoRA (shared random projection)
- ⚠️ Performance may be marginally lower than LoRA in some cases
When to use VeRA:
- Extreme memory constraints (mobile, edge devices)
- Fine-tuning many layers with limited resources
- Rapid prototyping with minimal parameter overhead
- When LoRA is still too expensive
How It Works
VeRA achieves 10x fewer trainable parameters than standard LoRA by:
- Using a single pair of random low-rank matrices (A and B) shared across ALL layers
- Freezing these shared matrices (they are never trained)
- Training only small scaling vectors (d and b) that are specific to each layer
The forward computation is: output = base_layer(input) + d * (B * A * input) * b where d and b are trainable vectors, and A and B are frozen shared matrices.
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new VeRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured VeRAAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
VeRAAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) |
Initializes a new VeRA adapter wrapping an existing layer. |
Properties
| Property | Summary |
|---|---|
bool AreSharedMatricesInitialized { get; } |
Gets whether the shared matrices have been initialized. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the VeRA adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void InitializeSharedMatrices(int inputSize, int outputSize, int rank, int? seed) |
Initializes the shared random matrices used by all VeRA adapters. |
void ResetSharedMatrices() |
Resets the shared matrices (useful for testing or reinitializing). |
void ResetState() |
Resets the internal state of the VeRA adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |