ICMFusionAlgorithm<T, TInput, TOutput>
Models & Types · AiDotNet.MetaLearning.Algorithms
Implementation of ICM-Fusion: In-Context Meta-Optimized LoRA Fusion (Shao et al., AAAI 2026).
Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → ICMFusionAlgorithm<T, TInput, TOutput>
Implements: IGeneratedParameterRegistrar<T>
For Beginners
Fine-tuning a model on one task gives a small set of changes to its weights (an adapter). Fine-tuning on another task gives different changes, and simply averaging the two often makes the model worse at both. ICM-Fusion trains a small variational autoencoder to squeeze each adapter into a short code, blends the codes, and turns the blend back into one adapter that works for both tasks. The original weights are never overwritten: the model you get is always “original weights + blended adapter”.
How It Works
ICM-Fusion fuses task-specific adapters through a Fusion VAE (F-VAE). The pretrained weights stay frozen;
what is encoded, fused and reconstructed is the ADAPTER - the offset a task’s fine-tuning puts on top of
them, which is what a LoRA is (W = W_pt + BA). Here the adapter of task i is
l_i = theta_i - theta_pt, where theta_pt is the meta-model’s parameters at construction.
Meta-training, Algorithm 1 of the paper, per task:
Gradients are exact, not estimated. The encoder and decoder are affine, so L_meta is differentiated in
closed form, reparameterization included. Step 9 makes l_adapt depend on the decoder and, through z_i, on the
encoder; with UseFirstOrder = false that dependence is back-propagated through every refinement step as
(I - beta H) products, each Hessian-vector product taken as a central difference of task gradients.
With UseFirstOrder = true (the default) l_adapt is held constant, exactly as first-order MAML holds the
adapted weights constant.
Beyond the paper: the meta-model is the fused model. The paper fuses at inference only. Here, after each
VAE update, the batch’s posterior means are pushed into a decaying component history (NumFusionComponents,
FusionDecay), averaged in latent space, decoded, and written back as
theta = theta_pt + D([z_fused; v_fused]). The decoder starts at zero - the LoRA convention (Hu et al. 2022)
that an untrained adapter is the identity - so the meta-model starts exactly at the pretrained weights and moves
only as far as the VAE has learned to reconstruct real adapters. The next batch then fine-tunes from the fused
model, so the history accumulates rather than restarting.
Parameter vectors wider than 128 are compressed for the VAE into contiguous blocks: a block’s value is the mean of its entries, a decoded block is written to all of its entries, and a gradient reaches a block as the sum over its entries - the chain rule of that write.
Constructors
| Constructor | Summary |
|---|---|
ICMFusionAlgorithm(ICMFusionOptions<T, TInput, TOutput> options) |
Properties
| Property | Summary |
|---|---|
MetaLearningAlgorithmType AlgorithmType { get; } |
Methods
| Method | Summary |
|---|---|
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task) |
|
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch) |