CTCSegmentation<T>
Models & Types · AiDotNet.SpeechRecognition.CTCVariants
CTC-segmentation (Kurzinger et al. 2020): audio-text alignment from the CTC posteriors of a hybrid CTC/attention Transformer.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CTCSegmentation<T>
Implements: ISpeechRecognizer<T>
For Beginners
CTC-segmentation turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Conformer blocks, each mixing self-attention (long-range context) with a convolution (local patterns). A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.
How It Works
The network is the hybrid CTC/attention Transformer the paper aligns with (footnote 3: ESPnet v0.5.3’s default, a 12-layer self-attention encoder with 2048 feed-forward units, width 256 and 4 heads, a 6-layer decoder, CTC weight 0.3). Segment runs the paper’s algorithm (section 3.1): the trellis of maximum joint probabilities over CTC posteriors (Equation 1), where staying before the first character costs nothing so the transcript may start anywhere, backtracking from the most probable end of the last character (Equation 2), and a confidence per segment that is the minimum over windows of L frames of the mean frame probability (Equation 3). SpecAugment masks the features in training (section 3.1). Training minimizes the joint CTC/attention loss (Watanabe et al. 2017); transcription decodes the attention decoder greedily. The paper’s blocks use relative
positional encoding; these use rotary position encoding, the relative scheme
ConformerBlockLayer provides.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 34);
var model = new CTCSegmentation<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
CTCSegmentation(NeuralNetworkArchitecture<T> architecture, CTCSegmentationOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CTCSegmentation(NeuralNetworkArchitecture<T> architecture, string modelPath, CTCSegmentationOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the attention decoder with greedy decoding. |
ValueTuple<int[], double> Segment(Tensor<T> audio, IReadOnlyList<int> labels) |
Aligns a transcript to audio by CTC-segmentation (Kurzinger et al. |
string DetectLanguage(Tensor<T> audio) |