MERT<T>

Models & Types · AiDotNet.Audio.Foundations

MERT self-supervised music understanding foundation model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → MERT<T>

Implements: IAudioFoundationModel<T>

For Beginners

MERT is like HuBERT but for music. It deeply understands musical structure - pitch, harmony, rhythm, instrumentation - without needing labeled data. Use it as a feature extractor: feed it music and get embeddings useful for genre classification, instrument detection, mood analysis, and more.

Usage:

How It Works

MERT (Li et al., 2024) uses acoustic and musical tokenizers to learn rich music representations. Unlike speech models, it incorporates music-specific knowledge through CQT-based teacher targets and codebook clustering, enabling strong performance on 14 music information retrieval tasks including tagging, genre, instrument, and key detection.

Constructors

Constructor Summary
MERT(NeuralNetworkArchitecture<T> architecture, MERTOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a MERT model in native training mode.
MERT(NeuralNetworkArchitecture<T> architecture, string modelPath, MERTOptions options) Creates a MERT model in native training mode.

Properties

Property Summary
int EmbeddingDimension { get; }
int NumLayers { get; }

Methods

Method Summary
ModelMetadata<T> GetModelMetadata()
Task<Tensor<T>> ExtractEmbeddingsAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbeddings(Tensor<T> audio)
Tensor<T> ExtractLayerFeatures(Tensor<T> audio, int layerIndex)
Tensor<T> ExtractWeightedFeatures(Tensor<T> audio, T[] layerWeights)