CIFDecoder<T>

Models & Types · AiDotNet.SpeechRecognition.CTCVariants

The CIF-based recognizer of Dong and Xu (2020): continuous integrate-and-fire between a self-attention encoder and a non-autoregressive decoder.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CIFDecoder<T>

Implements: ISpeechRecognizer<T>

For Beginners

Speech has far more audio frames than words. CIF learns a weight for every encoded frame, adds the weights up as it moves through the audio, and each time the running total reaches one it “fires”, handing the decoder one summary of the frames that made up that label. The decoder then writes every label at once.

How It Works

The network (CifAsrNetwork) is the paper’s section 3.3 with the sizes of section 4: the self-attention aligner’s encoder (a strided convolution and a multiplicative unit over filterbanks with deltas, then five self-attention layers at each of three time resolutions with pooling between them, encoding at 1/8 of the frame rate), a convolutional weight predictor, CIF with threshold 1, and a two-layer non-autoregressive decoder. Training scales the weights to the target length and minimizes the label-smoothed (0.2) cross-entropy on the targets ending with , plus 0.5 times the encoder’s CTC loss and the quantity loss; inference fires once more when the residual weight exceeds 0.5. Speed perturbation, SpecAugment, dropout, scheduled sampling, beam search and the language model rescoring are not applied. The paper states no optimizer; AdamW at 2e-4 is the default.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 16000, outputSize: 5230);
var model = new CIFDecoder<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
CIFDecoder(NeuralNetworkArchitecture<T> architecture, CIFDecoderOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
CIFDecoder(NeuralNetworkArchitecture<T> architecture, string modelPath, CIFDecoderOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: CIF over the encoder output, then one parallel decoder pass.
string DetectLanguage(Tensor<T> audio)