ICMFusionAlgorithm<T, TInput, TOutput>

Models & Types · AiDotNet.MetaLearning.Algorithms

Implementation of ICM-Fusion: In-Context Meta-Optimized LoRA Fusion (Shao et al., AAAI 2026).

Inheritance: ModelBase<T, TInput, TOutput> → MetaLearnerBase<T, TInput, TOutput> → ICMFusionAlgorithm<T, TInput, TOutput>

Implements: IGeneratedParameterRegistrar<T>

For Beginners

Fine-tuning a model on one task gives a small set of changes to its weights (an adapter). Fine-tuning on another task gives different changes, and simply averaging the two often makes the model worse at both. ICM-Fusion trains a small variational autoencoder to squeeze each adapter into a short code, blends the codes, and turns the blend back into one adapter that works for both tasks. The original weights are never overwritten: the model you get is always “original weights + blended adapter”.

How It Works

ICM-Fusion fuses task-specific adapters through a Fusion VAE (F-VAE). The pretrained weights stay frozen; what is encoded, fused and reconstructed is the ADAPTER - the offset a task’s fine-tuning puts on top of them, which is what a LoRA is (W = W_pt + BA). Here the adapter of task i is l_i = theta_i - theta_pt, where theta_pt is the meta-model’s parameters at construction.

Meta-training, Algorithm 1 of the paper, per task:

Gradients are exact, not estimated. The encoder and decoder are affine, so L_meta is differentiated in closed form, reparameterization included. Step 9 makes l_adapt depend on the decoder and, through z_i, on the encoder; with UseFirstOrder = false that dependence is back-propagated through every refinement step as (I - beta H) products, each Hessian-vector product taken as a central difference of task gradients. With UseFirstOrder = true (the default) l_adapt is held constant, exactly as first-order MAML holds the adapted weights constant.

Beyond the paper: the meta-model is the fused model. The paper fuses at inference only. Here, after each VAE update, the batch’s posterior means are pushed into a decaying component history (NumFusionComponents, FusionDecay), averaged in latent space, decoded, and written back as theta = theta_pt + D([z_fused; v_fused]). The decoder starts at zero - the LoRA convention (Hu et al. 2022) that an untrained adapter is the identity - so the meta-model starts exactly at the pretrained weights and moves only as far as the VAE has learned to reconstruct real adapters. The next batch then fine-tunes from the fused model, so the history accumulates rather than restarting.

Parameter vectors wider than 128 are compressed for the VAE into contiguous blocks: a block’s value is the mean of its entries, a decoded block is written to all of its entries, and a gradient reaches a block as the sum over its entries - the chain rule of that write.

Constructors

Constructor Summary
ICMFusionAlgorithm(ICMFusionOptions<T, TInput, TOutput> options)

Properties

Property Summary
MetaLearningAlgorithmType AlgorithmType { get; }

Methods

Method Summary
IModel<TInput, TOutput, ModelMetadata<T>> Adapt(IMetaLearningTask<T, TInput, TOutput> task)
T MetaTrain(TaskBatch<T, TInput, TOutput> taskBatch)