LoHaAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
LoHa (Low-Rank Hadamard Product Adaptation) adapter for parameter-efficient fine-tuning.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → LoHaAdapter<T>
For Beginners
LoHa is a variant of LoRA that uses element-wise multiplication instead of matrix multiplication. Think of it this way:
- Standard LoRA: Learns “row and column patterns” that combine via matrix multiply
- LoHa: Learns “pixel-by-pixel patterns” that combine via element-wise multiply
LoHa is especially good when:
- You need to capture local, element-wise patterns (like in images)
- The weight matrix has spatial structure (like convolutional filters)
- You want each weight to be adjusted somewhat independently
Trade-offs compared to LoRA:
- More parameters: Both A and B must be full-sized (input×output) per rank dimension
- Different expressiveness: Better for element-wise patterns, different from matrix patterns
- Better for CNNs: The element-wise nature matches convolutional structure better
Example: A 100×100 weight matrix with rank=8
- Standard LoRA: 8×100 + 100×8 = 1,600 parameters
- LoHa: 2 × 8 × 100 × 100 = 160,000 parameters (each rank has 2 full-sized matrices)
LoHa uses MORE parameters than LoRA but models element-wise weight interactions via Hadamard products.
How It Works
LoHa uses element-wise Hadamard products (⊙) instead of matrix multiplication for adaptation. Instead of computing ΔW = B * A like standard LoRA, LoHa computes: ΔW = sum over rank of (A[i] ⊙ B[i])
This formulation can capture element-wise patterns that matrix multiplication may miss, making it particularly effective for:
- Convolutional layers (local spatial patterns)
- Element-wise transformations
- Fine-grained weight adjustments
Mathematical Formulation:
Standard LoRA: ΔW = B * A where B is rank×output, A is input×rank LoHa: ΔW = Σ(A[i] ⊙ B[i]) where A[i] and B[i] are both input×output
The Hadamard product (⊙) performs element-wise multiplication, allowing each element of the weight matrix to be adjusted independently across the rank dimensions.
Example
using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;
var adapter = new LoHaAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured LoHaAdapter (rank {config.Rank}).");
Constructors
| Constructor | Summary |
|---|---|
LoHaAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) |
Initializes a new LoHa adapter wrapping an existing layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
ILayer<T> MergeToOriginalLayer() |
Merges the LoHa adaptation into the base layer and returns the merged layer. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ResetState() |
Resets the internal state of both the base layer and LoHa adapter. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using the specified learning rate. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |