SPIRAL<T>
Models & Types · AiDotNet.SpeechRecognition.Foundation
SPIRAL: a convolution-Transformer speech encoder pre-trained to make its representation of perturbed speech match a moving-average teacher’s representation of the clean speech, with a convolutional CTC classifier.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SPIRAL<T>
Implements: ISpeechRecognizer<T>
For Beginners
SPIRAL learns from unlabeled speech with two copies of one network. The teacher hears the clean recording; the student hears it with parts blanked out, and learns to produce what the teacher produced. The teacher is a slowly moving average of the student, so the two improve together. Afterwards the student’s encoder is fine-tuned on transcribed speech.
How It Works
References:
- Paper: “SPIRAL: Self-supervised Perturbation-Invariant Representation Learning for Speech Pre-Training” (Huang et al., ICLR 2022)
The network is the BASE model of Table 1: 128-bin log-mel input, two blocks of down-sampling convolutions and a
pre-norm Transformer with convolutional relative position encoding (40 ms, then 80 ms frames), a linear projection head
and the student’s convolutional predictor (section 3.2). Pretrain runs the pre-training of section 3.1:
adaptive SpecAugment on the student’s input (section 3.3), random padding of the teacher’s input against positional
collapse, the in-utterance contrastive loss of Equation 2, and the teacher’s moving-average update on a cosine
schedule (Equation 1, section 4.2). Ordinary training fine-tunes the encoder with the convolutional classifier and CTC
(section 3.5).
Example
// Create a SPIRAL recognizer
var architecture = new NeuralNetworkArchitecture<double>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputHeight: 16000, inputWidth: 1, inputDepth: 1, outputSize: 1024);
var model = new SPIRAL<double>(architecture);
// Or load an exported ONNX model
var onnxModel = new SPIRAL<double>(architecture, "spiral.onnx");
Constructors
| Constructor | Summary |
|---|---|
SPIRAL(NeuralNetworkArchitecture<T> architecture, SPIRALOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SPIRAL(NeuralNetworkArchitecture<T> architecture, string modelPath, SPIRALOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
T Pretrain(Tensor<T> audio) |
One SPIRAL pre-training step (section 3.1) on waveforms [S] or [batch, S]: a gradient step on the student, then the teacher’s moving-average update with α on the cosine schedule from TeacherMomentumStart to TeacherMomentumEnd over Pretr… |
T PretrainingLoss(Tensor<T> audio, Random random) |
The SPIRAL pre-training loss of waveforms [S] or [batch, S] (Huang et al. |
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the encoder, projection and CTC layer give per-frame distributions, decoded greedily. |
string DetectLanguage(Tensor<T> audio) |