LoRALayer<T>

Layers · AiDotNet.LoRA

Implements Low-Rank Adaptation (LoRA) layer for parameter-efficient fine-tuning of neural networks.

Inheritance: LayerBase<T> → LoRALayer<T>

Implements: IShapeContract

For Beginners

LoRA is a technique that makes it much cheaper to adapt large neural networks to new tasks. Instead of updating all the weights in a layer (which can be millions of parameters), LoRA adds two small matrices that work together to approximate the needed changes.

Think of it like this:

  • Traditional fine-tuning: Adjusting every single knob on a massive control panel
  • LoRA: Using just a few master controls that influence many knobs at once

The key insight is that the changes needed for fine-tuning often lie in a “low-rank” space, meaning we don’t need full freedom to adjust every parameter independently.

Key parameters:

  • Rank (r): Controls how many “master controls” you have. Higher rank = more flexibility but more parameters
  • Alpha: A scaling factor that controls how much influence the LoRA adaptation has

For example, adapting a layer with 1000x1000 weights (1M parameters) using LoRA with rank=8 only requires 8x1000 + 8x1000 = 16,000 parameters (98.4% reduction!).

How It Works

LoRA works by decomposing weight updates into two low-rank matrices A and B, where the actual update is computed as B * A. This dramatically reduces the number of trainable parameters compared to fine-tuning all weights directly.

Constructors

Constructor Summary
LoRALayer(int inputSize, int outputSize, int rank, double alpha, IActivationFunction<T> activationFunction) Initializes a new LoRA layer with the specified dimensions and hyperparameters.

Properties

Property Summary
T Alpha { get; } Gets the alpha scaling factor.
T Scaling { get; } Gets the computed scaling factor (alpha / rank).
bool SupportsTraining { get; } Gets whether this layer supports training (always true for LoRA).
int Rank { get; } Gets the rank of this LoRA layer.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Matrix<T> GetMatrixA() Gets matrix A (for inspection or advanced use cases).
Matrix<T> GetMatrixB() Gets matrix B (for inspection or advanced use cases).
Matrix<T> MergeWeights() Merges the LoRA weights into a dense weight matrix that can be added to a base layer.
void ResetState() Resets the internal state of the layer.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the layer’s parameters using the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).