SelfConditionedCTC<T>

Models & Types · AiDotNet.SpeechRecognition.CTCVariants

Self-conditioned CTC (Nozaki and Komatsu 2021): intermediate CTC predictions fed back into the encoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SelfConditionedCTC<T>

Implements: ISpeechRecognizer<T>

For Beginners

Self-conditioned CTC turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Transformer layers. A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.

How It Works

The encoder is the paper’s 18-layer Transformer (width 256, 4 heads, 2048 feed-forward units, section 3.2). At every third layer (K = 5, layers floor(kL/(K + 1))) the shared layer norm and CTC layer give an intermediate prediction, which a shared linear layer adds back to the next layer’s input (Equation 9); training minimizes (1 - lambda) L_CTC + lambda L_inter with lambda = 0.5 (Equation 12). SpecAugment masks the features in training. The CTC layer trains with the CTC loss (Graves et al. 2006) and decodes greedily. Positions are absolute sinusoids added to the scaled input, as in ESPnet’s Transformer encoder.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 34);
var model = new SelfConditionedCTC<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
SelfConditionedCTC(NeuralNetworkArchitecture<T> architecture, SelfConditionedCTCOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SelfConditionedCTC(NeuralNetworkArchitecture<T> architecture, string modelPath, SelfConditionedCTCOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the encoder and the CTC layer with greedy decoding.
string DetectLanguage(Tensor<T> audio)