LoKrAdapter<T>

Models & Types · AiDotNet.LoRA.Adapters

LoKr (Low-Rank Kronecker Product Adaptation) adapter for parameter-efficient fine-tuning.

Inheritance: LayerBase<T> → LoRAAdapterBase<T> → LoKrAdapter<T>

For Beginners

LoKr is a variant of LoRA that uses a different mathematical operation called the Kronecker product. Think of it this way:

  • Standard LoRA: Multiplies two small matrices (like 1000×8 and 8×1000) to approximate changes
  • LoKr: Uses Kronecker product of two even smaller matrices (like 50×4 and 20×4) to create the same size output

The Kronecker product creates a larger matrix by taking every element of the first matrix and multiplying it by the entire second matrix. This creates a block pattern that’s very efficient for representing certain types of structured transformations.

When to use LoKr vs standard LoRA:

  • LoKr is better for very wide or very deep layers (e.g., 10000×10000 weight matrices)
  • LoKr can achieve similar expressiveness with fewer parameters than LoRA
  • Standard LoRA is simpler and works well for typical layer sizes

Parameter Efficiency Example: For a 1000×1000 weight matrix with rank r=8:

  • Standard LoRA: 1000×8 + 8×1000 = 16,000 parameters
  • LoKr: 50×4 + 20×4 = 200 + 80 = 280 parameters (57x fewer!)

(where 50×20 = 1000 for both dimensions)

How It Works

LoKr uses Kronecker products instead of standard matrix multiplication for low-rank adaptation. Instead of computing ΔW = A × B (standard LoRA), LoKr computes ΔW = A ⊗ B where ⊗ is the Kronecker product. This is particularly efficient for very large weight matrices.

Kronecker Product Definition: For matrices A (m×n) and B (p×q), the Kronecker product A ⊗ B is an (m×p) × (n×q) matrix:

A ⊗ B = [a₁₁B a₁₂B … a₁ₙB] [a₂₁B a₂₂B … a₂ₙB] [ ⋮ ⋮ ⋱ ⋮ ] [aₘ₁B aₘ₂B … aₘₙB]

Each element aᵢⱼ of A is multiplied by the entire matrix B, creating a block structure.

Example

using AiDotNet.LoRA;
using AiDotNet.LoRA.Adapters;

var adapter = new LoKrAdapter<double>(null, rank: 8, alpha: 8, freezeBaseLayer: true);
var config = new DefaultLoRAConfiguration<double>(rank: 8, alpha: 8, loraAdapter: adapter);
Console.WriteLine($"Configured LoKrAdapter (rank {config.Rank}).");

Constructors

Constructor Summary
LoKrAdapter(ILayer<T> baseLayer, int rank, double alpha, bool freezeBaseLayer) Initializes a new LoKr adapter wrapping an existing layer.

Properties

Property Summary
ValueTuple<int, int> MatrixADimensions { get; } Gets the dimensions of matrix A.
ValueTuple<int, int> MatrixBDimensions { get; } Gets the dimensions of matrix B.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
ILayer<T> MergeToOriginalLayer() Merges the LoKr adaptation into the base layer and returns the merged layer.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Matrix<T> GetMatrixA() Gets matrix A (for inspection or advanced use cases).
Matrix<T> GetMatrixB() Gets matrix B (for inspection or advanced use cases).
void ResetState() Resets the internal state of the adapter.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the layer’s parameters using the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).