FireRedASRLLM<T>

Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated

FireRedASR-LLM (FireRed Team, 2025): the Encoder-Adapter-LLM recognizer of FireRedASR, which feeds the FireRedASR-AED Conformer encoder’s output through an adapter into Qwen2-7B-Instruct.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → FireRedASRLLM<T>

Implements: ISpeechRecognizer<T>

For Beginners

FireRedASR-LLM listens with the encoder of FireRedASR’s encoder-decoder model, shrinks its output to half the frames, and hands it to a large language model (Qwen2-7B-Instruct) as part of a chat message that asks for a transcript. The language model then writes the transcript.

How It Works

§2.2 and Figure 1 of the paper: the encoder is initialized from FireRedASR-AED’s; the adapter splices adjacent frames and applies a linear layer with ReLU and a linear layer into the LLM’s embedding space; the LLM is Qwen2-7B-Instruct. The release (fireredasr_llm.py, adapter.py, llm_tokenizer.py) fixes the details: two frames per adapter step with the remainder discarded; the prompt <|im_start|>user\n<speech>请转写音频为文字 <|im_end|>\n<|im_start|>assistant\n with the adapter’s output in place of <speech>; and LoRA of rank 64 (alpha 16, dropout 0.05) on every attention and feed-forward projection. Training minimizes the cross-entropy over the transcript only. The paper trains the encoder, the adapter and the LoRA adapters with the LLM fixed (EncoderAdapterAndLora); a model built here starts from random weights, so the default updates every parameter. The paper states no optimizer; AdamW with the library’s defaults is used.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 160000, outputSize: 152064);
var model = new FireRedASRLLM<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
FireRedASRLLM(NeuralNetworkArchitecture<T> architecture, FireRedASRLLMOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
FireRedASRLLM(NeuralNetworkArchitecture<T> architecture, string modelPath, FireRedASRLLMOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the LLM’s greedy answer to the transcription prompt, at most as many tokens as the adapter emits frames (the release’s max_new_tokens) and MaxTextLength.
string DetectLanguage(Tensor<T> audio)