LoRA

41 public types in this namespace, organized by kind.

Models & Types (34)

Type Summary
AdaLoRAAdapter<T> Adaptive Low-Rank Adaptation (AdaLoRA) adapter that dynamically allocates parameter budgets among weight matrices.
ChainLoRAAdapter<T> Chain-of-LoRA adapter that implements sequential composition of multiple LoRA adapters.
DVoRAAdapter<T> DVoRA (DoRA + VeRA) adapter - combines DoRA’s magnitude-direction decomposition with VeRA’s extreme parameter efficiency.
DeltaLoRAAdapter<T> Delta-LoRA adapter that focuses on parameter-efficient delta updates with momentum.
DenseLoRAAdapter<T> LoRA adapter specifically for Dense and FullyConnected layers with 1D input/output shapes.
DoRAAdapter<T> DoRA (Weight-Decomposed Low-Rank Adaptation) adapter for parameter-efficient fine-tuning with improved stability.
DyLoRAAdapter<T> DyLoRA (Dynamic LoRA) adapter that trains with multiple ranks simultaneously.
FloraAdapter<T> Implements Flora (Low-Rank Adapters Are Secretly Gradient Compressors) adapter for memory-efficient fine-tuning.
GLoRAAdapter<T> Generalized LoRA (GLoRA) implementation that adapts both weights AND activations.
GraphConvolutionalLoRAAdapter<T> LoRA adapter for Graph Convolutional layers, enabling parameter-efficient fine-tuning of GNN models.
HRAAdapter<T> HRA (Hybrid Rank Adaptation) adapter that combines low-rank and full-rank updates for optimal parameter efficiency.
LoHaAdapter<T> LoHa (Low-Rank Hadamard Product Adaptation) adapter for parameter-efficient fine-tuning.
LoKrAdapter<T> LoKr (Low-Rank Kronecker Product Adaptation) adapter for parameter-efficient fine-tuning.
LoRADropAdapter<T> LoRA-drop implementation: LoRA with dropout regularization.
LoRAFAAdapter<T> LoRA-FA (LoRA with Frozen A matrix) adapter for parameter-efficient fine-tuning.
LoRAPlusAdapter<T> LoRA+ adapter that uses optimized learning rates for faster convergence and better performance.
LoRAXSAdapter<T> LoRA-XS (Extremely Small) adapter for ultra-parameter-efficient fine-tuning using SVD with trainable scaling matrix.
LoRETTAAdapter<T> LoRETTA (Low-Rank Economic Tensor-Train Adaptation) adapter for parameter-efficient fine-tuning.
LoftQAdapter<T> LoftQ (LoRA-Fine-Tuning-Quantized) adapter that combines quantization and LoRA with improved initialization.
LongLoRAAdapter<T> LongLoRA adapter that efficiently extends LoRA to handle longer context lengths using shifted sparse attention.
MoRAAdapter<T> Implements MoRA (High-Rank Updating for Parameter-Efficient Fine-Tuning) adapter.
MultiLoRAAdapter<T> Multi-task LoRA adapter that manages multiple task-specific LoRA layers for complex multi-task learning scenarios.
NOLAAdapter<T> Implements NOLA (Compressing LoRA using Linear Combination of Random Basis) adapter for extreme parameter efficiency.
PiSSAAdapter<T> Principal Singular Values and Singular Vectors Adaptation (PiSSA) adapter for parameter-efficient fine-tuning.
QALoRAAdapter<T> Quantization-Aware LoRA (QA-LoRA) adapter that combines parameter-efficient fine-tuning with group-wise quantization awareness.
QLoRAAdapter<T> QLoRA (Quantized LoRA) adapter for parameter-efficient fine-tuning with 4-bit quantized base weights.
ReLoRAAdapter<T> Restart LoRA (ReLoRA) adapter that periodically merges and restarts LoRA training for continual learning.
RoSAAdapter<T> RoSA (Robust Adaptation) adapter for parameter-efficient fine-tuning with improved robustness to distribution shifts.
SLoRAAdapter<T> S-LoRA adapter for scalable serving of thousands of concurrent LoRA adapters.
StandardLoRAAdapter<T> Standard LoRA implementation (original LoRA algorithm).
TiedLoRAAdapter<T> Tied-LoRA adapter - LoRA with weight tying for extreme parameter efficiency across deep networks.
VBLoRAAdapter<T> Vector Bank LoRA (VB-LoRA) adapter that uses shared parameter banks for efficient multi-client deployment.
VeRAAdapter<T> VeRA (Vector-based Random Matrix Adaptation) adapter - an extreme parameter-efficient variant of LoRA.
XLoRAAdapter<T> X-LoRA (Mixture of LoRA Experts) adapter that uses multiple LoRA experts with learned routing.

Layers (1)

Type Summary
LoRALayer<T> Implements Low-Rank Adaptation (LoRA) layer for parameter-efficient fine-tuning of neural networks.

Base Classes (1)

Type Summary
LoRAAdapterBase<T> Abstract base class for LoRA (Low-Rank Adaptation) adapters that wrap existing layers.

Enums (2)

Type Summary
QuantizationType<T> Specifies the type of 4-bit quantization to use for base layer weights.
QuantizationType<T> Specifies the type of 4-bit quantization to use for base layer weights.

Options & Configuration (1)

Type Summary
DefaultLoRAConfiguration<T> Default LoRA configuration that applies LoRA to all layers with trainable weight matrices.

Helpers & Utilities (2)

Type Summary
LoRAAdapterMerger Bakes LoRA adapter layers into plain layers across a whole network, producing a model that serves the fine-tuned behavior with no per-layer LoRA overhead — the standard way to deploy a LoRA-fine-tuned model as its own servable variant.
LoRAAdapterSelection Shared helper for selecting the active multi-LoRA task across a network’s MultiLoRAAdapter layers.