LoRETTAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
LoRETTA (Low-Rank Economic Tensor-Train Adaptation) adapter for parameter-efficient fine-tuning.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → LoRETTAAdapter<T>
For Beginners
LoRETTA is an advanced version of LoRA that uses “tensor-train decomposition”!
Standard LoRA uses two matrices (A and B) to approximate weight changes:
- Matrix A: Compresses input to rank dimensions
- Matrix B: Expands back to output dimensions
- Parameters: inputSize × rank + rank × outputSize
LoRETTA uses multiple small “cores” chained together:
- Instead of 2 large matrices, use many small tensors
- Each core captures local correlations
- The cores are “contracted” (multiplied in sequence)
- Can express more complex patterns with fewer parameters
Why tensor-train decomposition?
- More expressive: Can capture higher-order correlations
- More efficient: Fewer parameters than matrix factorization
- Better compression: Exploits structure in weight updates
- Scalable: Grows logarithmically with dimensions
Example parameter counts for 1000×1000 layer:
- Full update: 1,000,000 parameters
- Standard LoRA (rank=8): 16,000 parameters (98.4% reduction)
- LoRETTA (rank=4, 3 cores): ~6,000 parameters (99.4% reduction, even better!)
Key parameters:
- ttRank: Controls compression (like LoRA’s rank but more powerful)
- numCores: How many tensor cores in the chain (typically 3-5)
- alpha: Scaling factor for the adaptation strength
When to use LoRETTA:
- Maximum parameter efficiency needed
- Weight updates have higher-order structure
- You have very large layers to adapt
- Standard LoRA isn’t expressive enough at low ranks
Reference: Tensor-train decomposition: I. V. Oseledets, “Tensor-train decomposition,” SIAM J. Scientific Computing, 2011.
How It Works
LoRETTA extends LoRA by using tensor-train decomposition instead of simple matrix factorization. Instead of representing weight updates as W = A × B, LoRETTA uses a tensor-train decomposition that captures higher-order correlations with even fewer parameters.
Tensor-train decomposition represents a high-dimensional tensor as a sequence of lower-dimensional “cores” that are contracted together. For a weight matrix W of size (m × n), the tensor-train representation is:
W[i,j] = G1[i] × G2 × G3 × … × Gd[j]
where each core Gk has dimensions (r_{k-1} × n_k × r_k), and r_k are the TT-ranks. The boundary ranks are r_0 = r_d = 1.
Constructors
| Constructor | Summary |
|---|---|
LoRETTAAdapter(ILayer<T> baseLayer, int ttRank, int numCores, double alpha, bool freezeBaseLayer) |
Initializes a new LoRETTA adapter wrapping an existing layer. |
Properties
| Property | Summary |
|---|---|
int NumCores { get; } |
Gets the number of cores in the tensor-train. |
int TTRank { get; } |
Gets the tensor-train rank. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the LoRETTA adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
string GetParameterEfficiencyMetrics() |
Gets parameter efficiency metrics for this LoRETTA adapter. |
void ResetState() |
Resets the internal state of the adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |