CTCSegmentation<T>

Models & Types · AiDotNet.SpeechRecognition.CTCVariants

CTC-segmentation (Kurzinger et al. 2020): audio-text alignment from the CTC posteriors of a hybrid CTC/attention Transformer.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CTCSegmentation<T>

Implements: ISpeechRecognizer<T>

For Beginners

CTC-segmentation turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

The network is the hybrid CTC/attention Transformer the paper aligns with (footnote 3: ESPnet v0.5.3’s default, a 12-layer self-attention encoder with 2048 feed-forward units, width 256 and 4 heads, a 6-layer decoder, CTC weight 0.3). Segment runs the paper’s algorithm (section 3.1): the trellis of maximum joint probabilities over CTC posteriors (Equation 1), where staying before the first character costs nothing so the transcript may start anywhere, backtracking from the most probable end of the last character (Equation 2), and a confidence per segment that is the minimum over windows of L frames of the mean frame probability (Equation 3). SpecAugment masks the features in training (section 3.1). Training minimizes the joint CTC/attention loss (Watanabe et al. 2017); transcription decodes the attention decoder greedily. The paper’s blocks use relative positional encoding; these use rotary position encoding, the relative scheme ConformerBlockLayer provides.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new CTCSegmentation<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
CTCSegmentation(NeuralNetworkArchitecture<T> architecture, CTCSegmentationOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
CTCSegmentation(NeuralNetworkArchitecture<T> architecture, string modelPath, CTCSegmentationOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the attention decoder with greedy decoding.
ValueTuple<int[], double> Segment(Tensor<T> audio, IReadOnlyList<int> labels) Aligns a transcript to audio by CTC-segmentation (Kurzinger et al.
string DetectLanguage(Tensor<T> audio)