SpeakerLM<T>

Models & Types · AiDotNet.Audio.Speaker

SpeakerLM language-model-based speaker diarization and verification model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SpeakerRecognitionBase<T> → SpeakerLM<T>

Implements: ISpeakerEmbeddingExtractor<T>, ISpeakerVerifier<T>

For Beginners

SpeakerLM figures out “who said what” in a conversation by treating speaker changes like a language. Just as a language model predicts the next word, SpeakerLM predicts the next speaker turn. It learns patterns like “after person A speaks, person B usually responds” to improve accuracy.

Usage:

How It Works

SpeakerLM (2024) applies language modeling techniques to speaker embeddings for improved speaker diarization. It treats speaker turns as a sequence modeling problem, using a transformer-based language model over speaker embeddings to predict who speaks when, achieving improved DER on common benchmarks.

Constructors

Constructor Summary
SpeakerLM(NeuralNetworkArchitecture<T> architecture, SpeakerLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a SpeakerLM model in native training mode.
SpeakerLM(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeakerLMOptions options) Creates a SpeakerLM model in native training mode.

Properties

Property Summary
ISpeakerEmbeddingExtractor<T> EmbeddingExtractor { get; }
T DefaultThreshold { get; }
bool IsOnnxMode { get; }
double MinimumDurationSeconds { get; }

Methods

Method Summary
IReadOnlyList<Tensor<T>> ExtractEmbeddings(IReadOnlyList<Tensor<T>> audioSegments)
ModelMetadata<T> GetModelMetadata()
SpeakerProfile<T> Enroll(IReadOnlyList<Tensor<T>> enrollmentAudio)
SpeakerProfile<T> Enroll(Tensor<T> enrollmentAudio)
SpeakerProfile<T> UpdateProfile(SpeakerProfile<T> existingProfile, Tensor<T> newAudio)
SpeakerVerificationResult<T> Verify(Tensor<T> audio, Tensor<T> referenceEmbedding)
SpeakerVerificationResult<T> Verify(Tensor<T> audio, Tensor<T> referenceEmbedding, T threshold)
SpeakerVerificationResult<T> VerifyWithReferenceAudio(Tensor<T> audio, Tensor<T> referenceAudio)
T ComputeScore(Tensor<T> audio, Tensor<T> referenceEmbedding)
T ComputeSimilarity(Tensor<T> embedding1, Tensor<T> embedding2)
T GetThresholdForFAR(double targetFAR)
Task<SpeakerVerificationResult<T>> VerifyAsync(Tensor<T> audio, Tensor<T> referenceEmbedding, CancellationToken cancellationToken)
Task<Tensor<T>> ExtractEmbeddingAsync(Tensor<T> audio, CancellationToken cancellationToken)
Tensor<T> ExtractEmbedding(Tensor<T> audio)