GraniteSpeech<T>

Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated

Granite-speech (Saon et al., IBM 2025): a conformer CTC acoustic encoder, a window Q-Former speech modality adapter and a Granite LLM with LoRA adapters, which writes the transcript of the audio it is given in its chat prompt.

Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → GraniteSpeech<T>

Implements: ISpeechRecognizer<T>

For Beginners

Granite-speech listens with a conformer encoder, squeezes every 15 encoder frames into 3 vectors with a small query transformer, and places those vectors in a chat message to IBM’s Granite language model, which answers with the transcript.

How It Works

§II of the paper and the release (modeling_granite_speech.py, the processor and the model card) fix the details. The encoder reads 80-band log-mels in pairs of stacked frames (160 values, 50 per second) through conformer blocks with block self-attention of 200 frames and Shaw relative positions, and self-conditioned CTC at the middle layer. The projector pads the encoder frames to windows of 15 and reads each with a two-layer Q-Former’s 3 learned queries (5× downsampling), then maps them into the LLM’s width; they replace the <|audio|> tokens of the chat prompt. The LLM is granite-3.3-instruct with its embedding, residual, attention and logits multipliers and LoRA of rank 64 on the query and value projections. The paper trains the encoder with CTC (0.2 weight on the middle layer), then the projector and LoRA adapters on the frozen encoder and LLM with the transcript’s cross-entropy (GraniteSpeechTrainingStage); a model built here starts from random weights, so the default trains every parameter on the transcript.

Example

var architecture = new NeuralNetworkArchitecture<float>(
    inputType: InputType.OneDimensional,
    taskType: NeuralNetworkTaskType.SpeechRecognition,
    inputSize: 160000, outputSize: 49160);
var model = new GraniteSpeech<float>(architecture);
var text = model.Transcribe(waveform).Text;

Constructors

Constructor Summary
GraniteSpeech(NeuralNetworkArchitecture<T> architecture, GraniteSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer)
GraniteSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, GraniteSpeechOptions options)

Properties

Property Summary
IReadOnlyList<string> SupportedLanguages { get; }
bool SupportsStreaming { get; }
bool SupportsWordTimestamps { get; }

Methods

Method Summary
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio)
IStreamingTranscriptionSession<T> StartStreamingSession(string language)
ModelMetadata<T> GetModelMetadata()
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken)
Tensor<T> ForwardForTraining(Tensor<T> input)
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) Transcribes audio: the LLM’s greedy answer to the transcription prompt, at most MaxTextLength tokens.
string DetectLanguage(Tensor<T> audio)