ABINetMultiTaskLoss<T>

Models & Types · AiDotNet.LossFunctions

ABINet’s multi-task objective: the weighted sum of the vision, language and fusion losses.

Inheritance: LossFunctionBase<T> → ABINetMultiTaskLoss<T>

For Beginners

ABINet has three parts that each make their own guess at the text: one that looks at the image, one that reasons about language, and one that combines them. Rather than grading only the combined guess, this grades all three and adds the scores up, so every part gets told how it did.

How It Works

ABINet (Fang et al., CVPR 2021, arXiv:2103.06495) supervises all three of its branches at once. Paper Eq. 5:

where L_v is the vision model’s character loss, L_l the language model’s, and L_f the fused output’s. Each term is the same underlying character loss (cross-entropy with logits by default), just applied to a different branch’s prediction.

This matters for more than accuracy. The paper’s AUTONOMOUS principle blocks gradient flow from the language model back into the vision model, and that is only sound because L_v supervises the vision model directly. With a single fused loss the barrier leaves the vision encoder with no gradient source at all.

Constructors

Constructor Summary
ABINetMultiTaskLoss(LossFunctionBase<T> characterLoss, double visionLossWeight, double languageLossWeight) Creates the multi-task objective.

Properties

Property Summary
LossFunctionBase<T> CharacterLoss { get; } Gets the per-branch character loss this objective sums over.

Methods

Method Summary
T CalculateLoss(Vector<T> predicted, Vector<T> actual) The multi-task objective on the flat-vector surface: lambda_v*L_v + lambda_l*L_l + L_f.
Tensor<T> ComputeTapeLoss(Tensor<T> predicted, Tensor<T> target)
ValueTuple<T, Tensor<T>> CalculateLossAndGradientGpu(Tensor<T> predicted, Tensor<T> actual)