HtsatSpectrogramImageLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
HTS-AT’s reshape_wav2img (Chen et al. 2022): turns a log-mel spectrogram into the square image its Swin
Transformer reads, by stretching it to the image’s time budget and folding time into stacked frequency bands.
Inheritance: LayerBase<T> → HtsatSpectrogramImageLayer<T>
Implements: IShapeContract
For Beginners
A 10-second spectrogram is long and thin. Vision transformers want square pictures, so this slices the long strip into four pieces and stacks them on top of each other.
How It Works
With specSize = S and freqRatio = r, the target spectrogram is S·r frames by S/r mel
bins (HTS-AT: 1024 × 64 for S = 256, r = 4). A shorter input is stretched with bicubic interpolation
(align_corners = True, PyTorch’s cubic coefficient A = −0.75), first along time and then along
frequency, exactly as the reference calls F.interpolate. The [S·r, S/r] map is then cut into
r consecutive time chunks of S frames that are stacked along the frequency axis, giving an
S × S image whose rows are the r chunks’ mel bins and whose columns are time within a chunk.
Input: [batch, channels, frames, mels]. Output: [batch, channels, S, S]. Channels are
carried through independently, so the four mel views of feature fusion stay four images.
Constructors
| Constructor | Summary |
|---|---|
HtsatSpectrogramImageLayer(int specSize, int freqRatio) |
Creates the fold. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int TargetFrames { get; } |
Frames the spectrogram is stretched to: specSize · freqRatio. |
int TargetMelBins { get; } |
Mel bins the spectrogram is stretched to: specSize / freqRatio. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
void ResetState() |