HtsatFusionPatchEmbeddingLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
HTS-AT’s patch embedding with LAION-CLAP’s 2-D feature fusion (Wu et al. 2023 §3.3; reference
htsat.PatchEmbed with enable_fusion=True, fusion_type='aff_2d').
Inheritance: LayerBase<T> → HtsatFusionPatchEmbeddingLayer<T>
Implements: IShapeContract
For Beginners
A long recording is seen twice: once squeezed into the usual length (the big picture) and once as three full-detail excerpts. The layer cuts both into patches and blends the detail into the big picture where it helps.
How It Works
The input is four spectrogram images [batch, 4, S, S]: channel 0 is the global view (the whole clip
shrunk to the 10-second budget) and channels 1-3 are three 10-second local crops from the front, middle and
back of the clip. The global view goes through the ordinary patch projection, a p × p stride-p
convolution. For a clip that is longer than the budget, each local crop goes through a p × 3p
stride-(p, 3p) convolution, the three results are laid side by side along time, zero-padded (or cut) to
the global map’s width, and merged into the global map by attentional feature fusion
(AttentionalFeatureFusionLayer). The map is then flattened to tokens and LayerNorm’d.
Which samples are “longer” is the optional second input, a [batch] tensor of 0/1. Samples that are not
longer keep their global map untouched. As in the reference, a training batch with no longer sample marks one
random sample as longer so the fusion branch still receives a gradient.
Constructors
| Constructor | Summary |
|---|---|
HtsatFusionPatchEmbeddingLayer(int patchSize, int embedDim) |
Creates the fused patch embedding. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
void ResetState() |