VGGishAudioEmbedding<T>

Layers · AiDotNet.NeuralNetworks.Layers

VGGish audio embedding: the AudioSet convolutional network that turns a log-mel patch into a compact per-segment embedding.

Inheritance: LayerBase<T> → VGGishAudioEmbedding<T>

Implements: IShapeContract

For Beginners

Sound arrives as a long list of samples, which is not a useful shape for a network. The usual pipeline turns it into a picture — time along one axis, pitch along the other — and then reads that picture with an image network. This layer is the image network. Give it one such picture and it returns a short list of numbers summarising what was heard.

How It Works

What this is. Hershey et al., CNN Architectures for Large-Scale Audio Classification (ICASSP 2017), released as vggish in the TensorFlow AudioSet models. It is VGG with the last convolutional group dropped and the classifier replaced by a 128-wide embedding: four convolutional groups (64, 128, 256x2, 512x2) of 3x3 SAME-padded filters, each followed by a 2x2 stride-2 max pool, then flatten and three fully-connected layers. The final layer is linear — the embedding is taken pre-activation, exactly as the reference implementation does.

What it expects. A stabilised log-mel patch shaped [frames, mels] — the paper uses 96 frames x 64 mel bins, which is 0.96 s at a 25 ms window and 10 ms hop. Produce it with MelSpectrogram configured through VGGishMelSpectrogramDefaults; do not hand-roll it. A leading singleton channel is added here, so the convolutional stack sees [1, frames, mels].

Why a composite rather than loose layers. The children are registered with RegisterSubLayer, so the base class declares each one as a parameter slot under a stable identifier. Parameter count, flat get/set, chunked restore and checkpoint identity all follow from that single registration — there is no hand-written positional walk to drift out of step, which is the failure mode that produces “Expected N parameters, got M” on a clone. This is the same composition model as torch.nn.Module, with the addition that the slot identifiers are stable rather than positional, so reordering or renaming does not invalidate a checkpoint the way a PyTorch state_dict key would.

Constructors

Constructor Summary
VGGishAudioEmbedding(int conv1Filters, int conv2Filters, int conv3Filters, int conv4Filters, int fullyConnectedWidth, int embeddingSize) Builds the embedding network.

Properties

Property Summary
bool SupportsTraining { get; }
int EmbeddingSize { get; } Size of the embedding this layer produces.
int FullyConnectedWidth { get; } Width of the two hidden dense layers.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
void ClearGradients()
void ResetState()
void SetTrainingMode(bool isTraining)
void UpdateParameters(T learningRate)

Fields

Field Summary
int PaperConv1Filters Filter count of the first convolutional group in the published network.
int PaperConv2Filters Filter count of the second convolutional group in the published network.
int PaperConv3Filters Filter count of the third convolutional group (two layers) in the published network.
int PaperConv4Filters Filter count of the fourth convolutional group (two layers) in the published network.
int PaperEmbeddingSize Size of the published embedding.
int PaperFullyConnectedWidth Width of the two hidden fully-connected layers in the published network.
int PaperPatchFrames Frames per patch in the published front-end (0.96 s at a 10 ms hop).