CIFDecoder<T>
Models & Types · AiDotNet.SpeechRecognition.CTCVariants
The CIF-based recognizer of Dong and Xu (2020): continuous integrate-and-fire between a self-attention encoder and a non-autoregressive decoder.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → CIFDecoder<T>
Implements: ISpeechRecognizer<T>
For Beginners
Speech has far more audio frames than words. CIF learns a weight for every encoded frame, adds the weights up as it moves through the audio, and each time the running total reaches one it “fires”, handing the decoder one summary of the frames that made up that label. The decoder then writes every label at once.
How It Works
The network (CifAsrNetwork) is the paper’s section 3.3 with the sizes of section 4: the
self-attention aligner’s encoder (a strided convolution and a multiplicative unit over filterbanks with deltas, then
five self-attention layers at each of three time resolutions with pooling between them, encoding at 1/8 of the frame
rate), a convolutional weight predictor, CIF with threshold 1, and a two-layer non-autoregressive decoder. Training
scales the weights to the target length and minimizes the label-smoothed (0.2) cross-entropy on the targets ending with
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 16000, outputSize: 5230);
var model = new CIFDecoder<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
CIFDecoder(NeuralNetworkArchitecture<T> architecture, CIFDecoderOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
CIFDecoder(NeuralNetworkArchitecture<T> architecture, string modelPath, CIFDecoderOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: CIF over the encoder output, then one parallel decoder pass. |
string DetectLanguage(Tensor<T> audio) |