InterCTC<T>
Models & Types · AiDotNet.SpeechRecognition.CTCVariants
InterCTC (Lee and Watanabe 2021): a CTC encoder regularized by a second CTC loss on its middle layer.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → InterCTC<T>
Implements: ISpeechRecognizer<T>
For Beginners
InterCTC turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Transformer layers. A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.
How It Works
The encoder is ESPnet’s Transformer of the paper’s experiments (12 layers, width 256, 4 heads, 2048 feed-forward units). The middle layer (layer 6, floor(L/2), section 3) is read by the shared CTC layer, and training minimizes (1 - w) L_CTC + w L_InterCTC with w = 0.3 (Equation 14); decoding uses the last layer only. SpecAugment masks the features in training. The CTC layer trains with the CTC loss (Graves et al. 2006) and decodes greedily. Positions are absolute sinusoids added to the scaled input, as in ESPnet’s Transformer encoder.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 34);
var model = new InterCTC<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
InterCTC(NeuralNetworkArchitecture<T> architecture, InterCTCOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
InterCTC(NeuralNetworkArchitecture<T> architecture, string modelPath, InterCTCOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the CTC layer with greedy decoding. |
string DetectLanguage(Tensor<T> audio) |