SelfConditionedCTC<T>
Models & Types · AiDotNet.SpeechRecognition.CTCVariants
Self-conditioned CTC (Nozaki and Komatsu 2021): intermediate CTC predictions fed back into the encoder.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SelfConditionedCTC<T>
Implements: ISpeechRecognizer<T>
For Beginners
Self-conditioned CTC turns 16 kHz audio into an 80-band log-mel spectrogram, shortens it 4x with two convolutions, and encodes it with Transformer layers. A linear layer gives every frame a probability for each token; collapsing repeats and blanks gives the transcript.
How It Works
The encoder is the paper’s 18-layer Transformer (width 256, 4 heads, 2048 feed-forward units, section 3.2). At every third layer (K = 5, layers floor(kL/(K + 1))) the shared layer norm and CTC layer give an intermediate prediction, which a shared linear layer adds back to the next layer’s input (Equation 9); training minimizes (1 - lambda) L_CTC + lambda L_inter with lambda = 0.5 (Equation 12). SpecAugment masks the features in training. The CTC layer trains with the CTC loss (Graves et al. 2006) and decodes greedily. Positions are absolute sinusoids added to the scaled input, as in ESPnet’s Transformer encoder.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 34);
var model = new SelfConditionedCTC<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
SelfConditionedCTC(NeuralNetworkArchitecture<T> architecture, SelfConditionedCTCOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SelfConditionedCTC(NeuralNetworkArchitecture<T> architecture, string modelPath, SelfConditionedCTCOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder and the CTC layer with greedy decoding. |
string DetectLanguage(Tensor<T> audio) |