FireRedASRLLM<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
FireRedASR-LLM (FireRed Team, 2025): the Encoder-Adapter-LLM recognizer of FireRedASR, which feeds the FireRedASR-AED Conformer encoder’s output through an adapter into Qwen2-7B-Instruct.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → FireRedASRLLM<T>
Implements: ISpeechRecognizer<T>
For Beginners
FireRedASR-LLM listens with the encoder of FireRedASR’s encoder-decoder model, shrinks its output to half the frames, and hands it to a large language model (Qwen2-7B-Instruct) as part of a chat message that asks for a transcript. The language model then writes the transcript.
How It Works
§2.2 and Figure 1 of the paper: the encoder is initialized from FireRedASR-AED’s; the adapter splices adjacent frames
and applies a linear layer with ReLU and a linear layer into the LLM’s embedding space; the LLM is Qwen2-7B-Instruct.
The release (fireredasr_llm.py, adapter.py, llm_tokenizer.py) fixes the details: two frames per
adapter step with the remainder discarded; the prompt <|im_start|>user\n<speech>请转写音频为文字 <|im_end|>\n<|im_start|>assistant\n with the adapter’s output in place of <speech>; and
LoRA of rank 64 (alpha 16, dropout 0.05) on every attention and feed-forward projection. Training minimizes the
cross-entropy over the transcript only. The paper trains the encoder, the adapter and the LoRA adapters with the LLM
fixed (EncoderAdapterAndLora); a model built here starts from random weights, so
the default updates every parameter. The paper states no optimizer; AdamW with the library’s defaults is used.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 160000, outputSize: 152064);
var model = new FireRedASRLLM<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
FireRedASRLLM(NeuralNetworkArchitecture<T> architecture, FireRedASRLLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
FireRedASRLLM(NeuralNetworkArchitecture<T> architecture, string modelPath, FireRedASRLLMOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the LLM’s greedy answer to the transcription prompt, at most as many tokens as the adapter emits frames (the release’s max_new_tokens) and MaxTextLength. |
string DetectLanguage(Tensor<T> audio) |