InterCTC<T>

Models & Types · AiDotNet.SpeechRecognition.CTCVariants

InterCTC (Lee and Watanabe 2021): a CTC encoder regularized by a second CTC loss on its middle layer.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → InterCTC<T>

Implements: ISpeechRecognizer<T>

For Beginners

InterCTC turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Transformer layers. A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

The encoder is ESPnet’s Transformer of the paper’s experiments (12 layers, width 256, 4 heads, 2048 feed-forward units). The middle layer (layer 6, floor(L/2), section 3) is read by the shared CTC layer, and training minimizes (1 - w) L_CTC + w L_InterCTC with w = 0.3 (Equation 14); decoding uses the last layer only. SpecAugment masks the features in training. The CTC layer trains with the CTC loss (Graves et al. 2006) and decodes greedily. Positions are absolute sinusoids added to the scaled input, as in ESPnet’s Transformer encoder.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new InterCTC<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
InterCTC(NeuralNetworkArchitecture<T> architecture, InterCTCOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
InterCTC(NeuralNetworkArchitecture<T> architecture, string modelPath, InterCTCOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the CTC layer with greedy decoding.
string DetectLanguage(Tensor<T> audio)