SpeakerDiarizedASR<T>

Models & Types · AiDotNet.SpeechRecognition.Specialized

Speaker-Diarized ASR: who-spoke-what transcription

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SpeakerDiarizedASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Speaker-Diarized ASR jointly performs speech recognition and speaker diarization, producing timestamped transcriptions attributed to individual speakers. The model uses a shared Conformer encoder with a speaker-aware serialized output (SA-SOT) dec…

How It Works

References:

  • Paper: “Multi-talker ASR with Speaker-Aware Serialized Output Training” (2024)

Speaker-Diarized ASR jointly performs speech recognition and speaker diarization, producing timestamped transcriptions attributed to individual speakers. The model uses a shared Conformer encoder with a speaker-aware serialized output (SA-SOT) decoder that emits speaker change tokens alongside text tokens. Speaker embeddings from an auxiliary speaker encoder condition the decoder to distinguish between speakers. The system handles overlapping speech through multi-talker serialized output.

Example

// Create a Speaker-Diarized ASR model for who-spoke-what transcription
var architecture = new NeuralNetworkArchitecture<double>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 5000);
var model = new SpeakerDiarizedASR<double>(architecture);
            
// Or load a pre-trained ONNX model for speaker-attributed ASR
var onnxModel = new SpeakerDiarizedASR<double>(architecture, "speakerdiarized.onnx");

Constructors

Constructor Summary
SpeakerDiarizedASR(NeuralNetworkArchitecture<T> architecture, SpeakerDiarizedASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SpeakerDiarizedASR(NeuralNetworkArchitecture<T> architecture, string modelPath, SpeakerDiarizedASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio with speaker attribution using SA-SOT decoding.
string DetectLanguage(Tensor<T> audio)