SeedASR<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
Seed-ASR (Bai et al., 2024): ByteDance’s audio-conditioned LLM recognizer, which feeds the LUISE Conformer encoder’s output through a frame-splicing converter into a large language model.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SeedASR<T>
Implements: ISpeechRecognizer<T>
For Beginners
Seed-ASR listens with a large Conformer encoder (LUISE), joins every four of its 40 ms frames into one 160 ms speech token, and hands those tokens to a language model, which writes the transcript. Text placed before the speech (a conversation, a document) can help it with names and rare words.
How It Works
§3.2–3.4 and Figure 2 of the paper: LUISE is a 32-block Conformer of nearly 2 billion parameters over mel filterbanks,
emitting 40 ms frames, pretrained BERT-style on masked frame labels; the converter splices four consecutive frames
along the features and applies a linear projection into the LLM; SFT trains the encoder and the converter with the
LLM fixed, with the cross-entropy over the transcript’s tokens only; context SFT places context text before the
speech; inference is greedy. The paper does not publish LUISE’s width, the LLM (a mixture-of-experts model of over
ten billion parameters), its tokenizer or the optimizer: see SeedASROptions for the defaults chosen.
A model built here starts from random weights and a fixed random LLM cannot fit a transcript, so the default updates
every parameter; the paper’s scope is EncoderAdapterAndLora.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 160000, outputSize: 32000);
var model = new SeedASR<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
SeedASR(NeuralNetworkArchitecture<T> architecture, SeedASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
SeedASR(NeuralNetworkArchitecture<T> architecture, string modelPath, SeedASROptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the LLM’s greedy continuation of the context and the speech, at most MaxTextLength tokens. |
string DetectLanguage(Tensor<T> audio) |