MultiLoRAAdapter<T>
Models & Types · AiDotNet.LoRA.Adapters
Multi-task LoRA adapter that manages multiple task-specific LoRA layers for complex multi-task learning scenarios.
Inheritance: LayerBase<T> → LoRAAdapterBase<T> → MultiLoRAAdapter<T>
For Beginners
Think of MultiLoRA as having one teacher (the base layer) and multiple students (task-specific LoRA adapters), each specializing in different subjects.
In regular LoRA:
- You have one base layer (the teacher)
- One LoRA adapter (one student learning one subject)
- Output = base + lora_adaptation
In MultiLoRA:
- You have one base layer (the teacher)
- Multiple LoRA adapters (multiple students, each specializing in different tasks)
- Output = base + task_specific_lora_adaptation
This is powerful for:
- Multi-domain learning: Train on medical, legal, and technical documents simultaneously
- Multi-lingual models: One adapter per language
- Multi-task learning: Sentiment analysis, named entity recognition, question answering, etc.
- Continual learning: Add new tasks without forgetting old ones
Example use case:
- Base: Pre-trained language model
- Task 1: Sentiment analysis (rank=4)
- Task 2: Named entity recognition (rank=8)
- Task 3: Question answering (rank=16)
You can switch between tasks at runtime, and each task only trains its specific LoRA weights!
How It Works
MultiLoRA extends the basic LoRA concept to handle multiple tasks simultaneously within a single layer. Instead of having one LoRA adaptation, it maintains a dictionary of task-specific LoRA layers, with a routing mechanism to select the appropriate adapter for each task.
Key features:
- Multiple task-specific LoRA adapters sharing the same base layer
- Dynamic task switching during inference and training
- Per-task rank configuration for optimal parameter efficiency
- Shared base layer weights across all tasks
- Task-specific merging for deployment
Constructors
| Constructor | Summary |
|---|---|
MultiLoRAAdapter(ILayer<T> baseLayer, string defaultTaskName, int defaultRank, double alpha, bool freezeBaseLayer) |
Initializes a new Multi-LoRA adapter with an initial default task. |
Properties
| Property | Summary |
|---|---|
IReadOnlyDictionary<string, LoRALayer<T>> TaskAdapters { get; } |
Gets the dictionary of task-specific LoRA adapters. |
int NumberOfTasks { get; } |
Gets the number of tasks configured in this adapter. |
string CurrentTask { get; set; } |
Gets or sets the name of the currently active task. |
Methods
| Method | Summary |
|---|---|
ILayer<T> MergeTaskToLayer(string taskName) |
Merges a specific task’s LoRA weights into the base layer. |
ILayer<T> MergeToOriginalLayer() |
Merges the currently active task’s LoRA weights into the base layer. |
LoRALayer<T> GetTaskAdapter(string taskName) |
Gets the LoRA layer for a specific task. |
bool RemoveTask(string taskName) |
Removes a task and its associated LoRA adapter. |
int GetTaskRank(string taskName) |
Gets the rank of a specific task’s LoRA adapter. |
void AddTask(string taskName, int rank, double alpha) |
Adds a new task with its own LoRA adapter. |
void ResetState() |
Resets the internal state of all layers. |
void SetCurrentTask(string taskName) |
Sets the current task for subsequent forward/backward operations. |
void UpdateParameters(T learningRate) |
Updates parameters for the current task only. |