Phi4Audio<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
Phi-4-multimodal’s speech recognition (Abdin et al., Microsoft 2025): a conformer audio encoder and a projector that feed Phi-4-Mini, which carries the speech LoRA adapters (“Mixture of LoRAs”) and writes the transcript.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → Phi4Audio<T>
Implements: ISpeechRecognizer<T>
For Beginners
Phi-4-multimodal hears through an audio encoder that turns each 80 ms of speech into one vector, maps those vectors into the language model’s embedding space with a small two-layer network, and asks the Phi-4-Mini language model to “Transcribe the audio clip into text.” Speech-specific LoRA adapters adjust the language model while leaving its text abilities untouched.
How It Works
§2.2.1 and the release (speech_conformer_encoder.py, modeling_phi4mm.py, processing_phi4mm.py)
fix the details: SpeechLib log filterbanks (pre-emphasis 0.97, Hamming 400, 512-point FFT, 80 bands to 7,690 Hz); a
global mean-variance normalization; NeMo’s depthwise-striding convolutions (8× subsampling); 24 conformer blocks of
width 1,024 with GLU feed-forward blocks, T5 relative attention biases and a causal depthwise convolution, attending
fully within chunks of 500 frames; a GELU MLP to 3,072; and Phi-4-Mini with partial rotary embeddings, LongRoPE and the
speech LoRA (rank 320, alpha 640) on its fused qkv, o, gate_up and down projections. The prompt is
<|user|><|audio_1|>Transcribe the audio clip into text.<|end|><|assistant|> and the
answer ends with <|end|><|endoftext|>. §2.2.2 trains the encoder and projector on a frozen decoder,
then the projector and the speech LoRA (Phi4AudioTrainingStage); a model built here starts from random
weights, so the default trains every parameter.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 160000, outputSize: 200064);
var model = new Phi4Audio<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
Phi4Audio(NeuralNetworkArchitecture<T> architecture, Phi4AudioOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
Phi4Audio(NeuralNetworkArchitecture<T> architecture, string modelPath, Phi4AudioOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the decoder’s greedy answer to the ASR prompt, at most MaxTextLength tokens. |
string DetectLanguage(Tensor<T> audio) |