VeRAAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

VeRA (Vector-based Random Matrix Adaptation) adapter - an extreme parameter-efficient variant of LoRA.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → VeRAAdapter<T>

For Beginners

VeRA is an ultra-efficient version of LoRA for extreme memory constraints.

Think of the difference this way:

  • Standard LoRA: Each layer has its own pair of small matrices (A and B) that are trained
  • VeRA: ALL layers share the same random matrices (A and B) which are frozen. Only tiny

scaling vectors are trained per layer.

Example parameter comparison for a 1000x1000 layer with rank=8:

  • Full fine-tuning: 1,000,000 parameters
  • Standard LoRA (rank=8): 16,000 parameters (98.4% reduction)
  • VeRA (rank=8): ~1,600 parameters (99.84% reduction) - 10x fewer than LoRA!

Trade-offs:

  • ✅ Extreme parameter efficiency (10x fewer than LoRA)
  • ✅ Very low memory footprint
  • ✅ Shared matrices reduce storage when adapting many layers
  • ⚠️ Slightly less flexible than standard LoRA (shared random projection)
  • ⚠️ Performance may be marginally lower than LoRA in some cases

When to use VeRA:

  • Extreme memory constraints (mobile, edge devices)
  • Fine-tuning many layers with limited resources
  • Rapid prototyping with minimal parameter overhead
  • When LoRA is still too expensive

How It Works

VeRA achieves 10x fewer trainable parameters than standard LoRA by:

  • Using a single pair of random low-rank matrices (A and B) shared across ALL layers
  • Freezing these shared matrices (they are never trained)
  • Training only small scaling vectors (d and b) that are specific to each layer

The forward computation is: output = base_layer(input) + d * (B * A * input) * b where d and b are trainable vectors, and A and B are frozen shared matrices.

Example

using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;

var adapter = new VeRAAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured VeRAAdapter (rank {config.Rank}).");

Constructors

Constructor Summary
VeRAAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) Initializes a new VeRA adapter wrapping an existing layer.

Properties

Property Summary
bool AreSharedMatricesInitialized { get; } Gets whether the shared matrices have been initialized.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
ILayer<T> MergeToOriginalLayer() Merges the VeRA adaptation into the base layer and returns the merged layer.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void InitializeSharedMatrices(int inputSize, int outputSize, int rank, int? seed) Initializes the shared random matrices used by all VeRA adapters.
void ResetSharedMatrices() Resets the shared matrices (useful for testing or reinitializing).
void ResetState() Resets the internal state of the VeRA adapter.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates parameters using the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).