LoRALayer<T>
Layers · AiDotNet.LoRA
Implements Low-Rank Adaptation (LoRA) layer for parameter-efficient fine-tuning of neural networks.
Inheritance: LayerBase<T> → LoRALayer<T>
Implements: IShapeContract
For Beginners
LoRA is a technique that makes it much cheaper to adapt large neural networks to new tasks. Instead of updating all the weights in a layer (which can be millions of parameters), LoRA adds two small matrices that work together to approximate the needed changes.
Think of it like this:
- Traditional fine-tuning: Adjusting every single knob on a massive control panel
- LoRA: Using just a few master controls that influence many knobs at once
The key insight is that the changes needed for fine-tuning often lie in a “low-rank” space, meaning we don’t need full freedom to adjust every parameter independently.
Key parameters:
- Rank (r): Controls how many “master controls” you have. Higher rank = more flexibility but more parameters
- Alpha: A scaling factor that controls how much influence the LoRA adaptation has
For example, adapting a layer with 1000x1000 weights (1M parameters) using LoRA with rank=8 only requires 8x1000 + 8x1000 = 16,000 parameters (98.4% reduction!).
How It Works
LoRA works by decomposing weight updates into two low-rank matrices A and B, where the actual update is computed as B * A. This dramatically reduces the number of trainable parameters compared to fine-tuning all weights directly.
Constructors
| Constructor | Summary |
|---|---|
LoRALayer(int inputSize, int outputSize, int rank, double alpha, IActivationFunction<T> activationFunction) |
Initializes a new LoRA layer with the specified dimensions and hyperparameters. |
Properties
| Property | Summary |
|---|---|
T Alpha { get; } |
Gets the alpha scaling factor. |
T Scaling { get; } |
Gets the computed scaling factor (alpha / rank). |
bool SupportsTraining { get; } |
Gets whether this layer supports training (always true for LoRA). |
int Rank { get; } |
Gets the rank of this LoRA layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Matrix<T> GetMatrixA() |
Gets matrix A (for inspection or advanced use cases). |
Matrix<T> GetMatrixB() |
Gets matrix B (for inspection or advanced use cases). |
Matrix<T> MergeWeights() |
Merges the LoRA weights into a dense weight matrix that can be added to a base layer. |
void ResetState() |
Resets the internal state of the layer. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates the layer’s parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |