VGGishAudioEmbedding<T>
Layers · AiDotNet.NeuralNetworks.Layers
VGGish audio embedding: the AudioSet convolutional network that turns a log-mel patch into a compact per-segment embedding.
Inheritance: LayerBase<T> → VGGishAudioEmbedding<T>
Implements: IShapeContract
For Beginners
Sound arrives as a long list of samples, which is not a useful shape for a network. The usual pipeline turns it into a picture — time along one axis, pitch along the other — and then reads that picture with an image network. This layer is the image network. Give it one such picture and it returns a short list of numbers summarising what was heard.
How It Works
What this is. Hershey et al., CNN Architectures for Large-Scale Audio Classification
(ICASSP 2017), released as vggish in the TensorFlow AudioSet models. It is VGG with the
last convolutional group dropped and the classifier replaced by a 128-wide embedding: four
convolutional groups (64, 128, 256x2, 512x2) of 3x3 SAME-padded filters, each followed by a 2x2
stride-2 max pool, then flatten and three fully-connected layers. The final layer is
linear — the embedding is taken pre-activation, exactly as the reference implementation
does.
What it expects. A stabilised log-mel patch shaped [frames, mels] — the paper uses
96 frames x 64 mel bins, which is 0.96 s at a 25 ms window and 10 ms hop. Produce it with
MelSpectrogram configured through
VGGishMelSpectrogramDefaults; do not hand-roll it. A leading singleton channel is
added here, so the convolutional stack sees [1, frames, mels].
Why a composite rather than loose layers. The children are registered with
RegisterSubLayer, so the base class declares each one as a parameter
slot under a stable identifier. Parameter count, flat get/set, chunked restore and checkpoint
identity all follow from that single registration — there is no hand-written positional walk to
drift out of step, which is the failure mode that produces “Expected N parameters, got M” on a
clone. This is the same composition model as torch.nn.Module, with the addition that the
slot identifiers are stable rather than positional, so reordering or renaming does not
invalidate a checkpoint the way a PyTorch state_dict key would.
Constructors
| Constructor | Summary |
|---|---|
VGGishAudioEmbedding(int conv1Filters, int conv2Filters, int conv3Filters, int conv4Filters, int fullyConnectedWidth, int embeddingSize) |
Builds the embedding network. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int EmbeddingSize { get; } |
Size of the embedding this layer produces. |
int FullyConnectedWidth { get; } |
Width of the two hidden dense layers. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
void ClearGradients() |
|
void ResetState() |
|
void SetTrainingMode(bool isTraining) |
|
void UpdateParameters(T learningRate) |
Fields
| Field | Summary |
|---|---|
int PaperConv1Filters |
Filter count of the first convolutional group in the published network. |
int PaperConv2Filters |
Filter count of the second convolutional group in the published network. |
int PaperConv3Filters |
Filter count of the third convolutional group (two layers) in the published network. |
int PaperConv4Filters |
Filter count of the fourth convolutional group (two layers) in the published network. |
int PaperEmbeddingSize |
Size of the published embedding. |
int PaperFullyConnectedWidth |
Width of the two hidden fully-connected layers in the published network. |
int PaperPatchFrames |
Frames per patch in the published front-end (0.96 s at a 10 ms hop). |