SeedASR<T>

Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated

Seed-ASR (Bai et al., 2024): ByteDance’s audio-conditioned LLM recognizer, which feeds the LUISE Conformer encoder’s output through a frame-splicing converter into a large language model.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → SeedASR<T>

Implements: ISpeechRecognizer<T>

For Beginners

Seed-ASR listens with a large Conformer encoder (LUISE), joins every four of its 40 ms frames into one 160 ms speech token, and hands those tokens to a language model, which writes the transcript. Text placed before the speech (a conversation, a document) can help it with names and rare words.

How It Works

§3.2–3.4 and Figure 2 of the paper: LUISE is a 32-block Conformer of nearly 2 billion parameters over mel filterbanks, emitting 40 ms frames, pretrained BERT-style on masked frame labels; the converter splices four consecutive frames along the features and applies a linear projection into the LLM; SFT trains the encoder and the converter with the LLM fixed, with the cross-entropy over the transcript’s tokens only; context SFT places context text before the speech; inference is greedy. The paper does not publish LUISE’s width, the LLM (a mixture-of-experts model of over ten billion parameters), its tokenizer or the optimizer: see SeedASROptions for the defaults chosen. A model built here starts from random weights and a fixed random LLM cannot fit a transcript, so the default updates every parameter; the paper’s scope is EncoderAdapterAndLora.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 160000, outputSize: 32000);
var model = new SeedASR<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
SeedASR(NeuralNetworkArchitecture<T> architecture, SeedASROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
SeedASR(NeuralNetworkArchitecture<T> architecture, string modelPath, SeedASROptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the LLM’s greedy continuation of the context and the speech, at most MaxTextLength tokens.
string DetectLanguage(Tensor<T> audio)