HtsatFusionPatchEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

HTS-AT’s patch embedding with LAION-CLAP’s 2-D feature fusion (Wu et al. 2023 §3.3; reference htsat.PatchEmbed with enable_fusion=True, fusion_type='aff_2d').

Inheritance: LayerBase<T> → HtsatFusionPatchEmbeddingLayer<T>

Implements: IShapeContract

For Beginners

A long recording is seen twice: once squeezed into the usual length (the big picture) and once as three full-detail excerpts. The layer cuts both into patches and blends the detail into the big picture where it helps.

How It Works

The input is four spectrogram images [batch, 4, S, S]: channel 0 is the global view (the whole clip shrunk to the 10-second budget) and channels 1-3 are three 10-second local crops from the front, middle and back of the clip. The global view goes through the ordinary patch projection, a p × p stride-p convolution. For a clip that is longer than the budget, each local crop goes through a p × 3p stride-(p, 3p) convolution, the three results are laid side by side along time, zero-padded (or cut) to the global map’s width, and merged into the global map by attentional feature fusion (AttentionalFeatureFusionLayer). The map is then flattened to tokens and LayerNorm’d.

Which samples are “longer” is the optional second input, a [batch] tensor of 0/1. Samples that are not longer keep their global map untouched. As in the reference, a training batch with no longer sample marks one random sample as longer so the fusion branch still receives a gradient.

Constructors

Constructor Summary
HtsatFusionPatchEmbeddingLayer(int patchSize, int embedDim) Creates the fused patch embedding.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
void ResetState()