HtsatSpectrogramImageLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

HTS-AT’s reshape_wav2img (Chen et al. 2022): turns a log-mel spectrogram into the square image its Swin Transformer reads, by stretching it to the image’s time budget and folding time into stacked frequency bands.

Inheritance: LayerBase<T> → HtsatSpectrogramImageLayer<T>

Implements: IShapeContract

For Beginners

A 10-second spectrogram is long and thin. Vision transformers want square pictures, so this slices the long strip into four pieces and stacks them on top of each other.

How It Works

With specSize = S and freqRatio = r, the target spectrogram is S·r frames by S/r mel bins (HTS-AT: 1024 × 64 for S = 256, r = 4). A shorter input is stretched with bicubic interpolation (align_corners = True, PyTorch’s cubic coefficient A = −0.75), first along time and then along frequency, exactly as the reference calls F.interpolate. The [S·r, S/r] map is then cut into r consecutive time chunks of S frames that are stacked along the frequency axis, giving an S × S image whose rows are the r chunks’ mel bins and whose columns are time within a chunk.

Input: [batch, channels, frames, mels]. Output: [batch, channels, S, S]. Channels are carried through independently, so the four mel views of feature fusion stay four images.

Constructors

Constructor Summary
HtsatSpectrogramImageLayer(int specSize, int freqRatio) Creates the fold.

Properties

Property Summary
bool SupportsTraining { get; }
int TargetFrames { get; } Frames the spectrogram is stretched to: specSize · freqRatio.
int TargetMelBins { get; } Mel bins the spectrogram is stretched to: specSize / freqRatio.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
void ResetState()