LoRA & Fine-Tuning

50+

Every major parameter-efficient fine-tuning method in pure C#

Adapt large language models and vision transformers with minimal compute. 50+ parameter-efficient fine-tuning methods and strategies from classic LoRA to cutting-edge techniques like GaLore and MoRA. Train only 0.1-1% of parameters while matching full fine-tuning performance.

LLM Customization Domain Adaptation Task-Specific Models Low-Resource Languages Edge Deployment Instruction Tuning Style Transfer Medical AI

LoRA Variants

Core LoRA family with different rank adaptation strategies.

LoRA

Low-Rank Adaptation freezing pretrained weights and training rank decomposition matrices.

QLoRA

4-bit quantized LoRA reducing memory to fine-tune 65B models on single GPU.

DoRA

Weight-Decomposed LoRA separating magnitude and direction for better accuracy.

AdaLoRA

Adaptive budget allocation pruning less important singular values during training.

LoRA+

Different learning rates for A and B matrices improving convergence.

rsLoRA

Rank-stabilized LoRA with scaling factor for higher-rank training.

PiSSA

Principal Singular values and Singular vectors Adaptation for faster convergence.

Delta-LoRA

Update pretrained weights using delta of LoRA products.

Advanced PEFT Methods

Next-generation parameter-efficient methods beyond standard LoRA.

VeRA

Vector-based Random Matrix Adaptation sharing frozen random matrices across layers.

MoRA

Employing high-rank updating with same parameter budget as LoRA.

GaLore

Gradient Low-Rank Projection enabling full-parameter learning with LoRA-level memory.

LoftQ

LoRA-Fine-Tuning-aware Quantization for better quantized starting point.

LongLoRA

Efficient fine-tuning for long context with shifted sparse attention.

LoRAMoE

Mixture of LoRA experts for multi-task adaptation.

CorDA

Context-Oriented Decomposition Adaptation leveraging world knowledge.

Other PEFT Methods

Alternative approaches to parameter-efficient adaptation.

Prefix Tuning

Prepend trainable continuous prompts to transformer layers.

Prompt Tuning

Soft prompts in input embedding space for task adaptation.

IA3

Infused Adapter by Inhibiting and Amplifying Inner Activations.

Adapter (Houlsby)

Bottleneck adapter layers inserted between transformer sub-layers.

UniPELT

Unified framework combining LoRA, prefix tuning, and adapters.

LoRA fine-tuning with AiModelBuilder

using AiDotNet;

// Fine-tune with LoRA using AiModelBuilder
var result = await new AiModelBuilder<float, float[], float>()
    .ConfigureModel(new NeuralNetwork<float>(
        inputSize: 768, hiddenSize: 256, outputSize: 10))
    .ConfigureLoRA(new QLoRAConfig(
        rank: 16, alpha: 32, quantBits: 4))
    .ConfigureOptimizer(new AdamOptimizer<float>(lr: 2e-4f))
    .BuildAsync(features, labels);

var prediction = result.Predict(newSample);

Start building with LoRA & Fine-Tuning

All 50+ implementations are included free under Apache 2.0.

Install AiDotNet Browse All Features