DistilWhisper<T>
Models & Types · AiDotNet.SpeechRecognition.WhisperFamily
Distil-Whisper (Gandhi et al. 2023): Whisper’s encoder kept frozen with a 2-layer decoder distilled from the teacher’s.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → DistilWhisper<T>
Implements: ISpeechRecognizer<T>
For Beginners
Distil-Whisper listens to 16 kHz audio in 30-second windows turned into a log-mel spectrogram, encodes each window with a Transformer, and a second Transformer writes the transcript token by token while reading the encoded audio.
How It Works
The network is distil-large-v3: the teacher’s 32-layer, 1280-wide encoder and a decoder of 2 layers initialised from the teacher’s first and last (section 3.2), which keeps Whisper’s architecture, tokenizer and decoding. The paper trains it with pseudo-label cross-entropy and KL divergence to the teacher (section 3.3); Train applies the cross-entropy term to the transcript it is given. Every forward reads the 30-second window openai-whisper builds (pad_or_trim); Predict returns the
decoder’s logits over the start sequence (start of transcript, language, transcribe, no timestamps), whose last row is
the first token’s distribution, and Train minimizes the teacher-forced cross-entropy of a transcript given as
token ids or as one distribution per token.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 480000, outputSize: 51866);
var model = new DistilWhisper<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
DistilWhisper(NeuralNetworkArchitecture<T> architecture, DistilWhisperOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
DistilWhisper(NeuralNetworkArchitecture<T> architecture, string modelPath, DistilWhisperOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes 16 kHz audio with openai-whisper’s transcribe: 30-second windows, temperature fallback, and word timing when timestamps are requested. |
string DetectLanguage(Tensor<T> audio) |