GraniteSpeech<T>
Models & Types · AiDotNet.SpeechRecognition.LLMIntegrated
Granite-speech (Saon et al., IBM 2025): a conformer CTC acoustic encoder, a window Q-Former speech modality adapter and a Granite LLM with LoRA adapters, which writes the transcript of the audio it is given in its chat prompt.
Inheritance: NeuralNetworkBase<T> → AudioNeuralNetworkBase<T> → GraniteSpeech<T>
Implements: ISpeechRecognizer<T>
For Beginners
Granite-speech listens with a conformer encoder, squeezes every 15 encoder frames into 3 vectors with a small query transformer, and places those vectors in a chat message to IBM’s Granite language model, which answers with the transcript.
How It Works
§II of the paper and the release (modeling_granite_speech.py, the processor and the model card) fix the
details. The encoder reads 80-band log-mels in pairs of stacked frames (160 values, 50 per second) through conformer
blocks with block self-attention of 200 frames and Shaw relative positions, and self-conditioned CTC at the middle
layer. The projector pads the encoder frames to windows of 15 and reads each with a two-layer Q-Former’s 3 learned
queries (5× downsampling), then maps them into the LLM’s width; they replace the <|audio|> tokens of
the chat prompt. The LLM is granite-3.3-instruct with its embedding, residual, attention and logits multipliers and
LoRA of rank 64 on the query and value projections. The paper trains the encoder with CTC (0.2 weight on the middle
layer), then the projector and LoRA adapters on the frozen encoder and LLM with the transcript’s cross-entropy
(GraniteSpeechTrainingStage); a model built here starts from random weights, so the default trains
every parameter on the transcript.
Example
var architecture = new NeuralNetworkArchitecture<float>(
inputType: InputType.OneDimensional,
taskType: NeuralNetworkTaskType.SpeechRecognition,
inputSize: 160000, outputSize: 49160);
var model = new GraniteSpeech<float>(architecture);
var text = model.Transcribe(waveform).Text;
Constructors
| Constructor | Summary |
|---|---|
GraniteSpeech(NeuralNetworkArchitecture<T> architecture, GraniteSpeechOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
|
GraniteSpeech(NeuralNetworkArchitecture<T> architecture, string modelPath, GraniteSpeechOptions options) |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<string> SupportedLanguages { get; } |
|
bool SupportsStreaming { get; } |
|
bool SupportsWordTimestamps { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyDictionary<string, T> DetectLanguageProbabilities(Tensor<T> audio) |
|
IStreamingTranscriptionSession<T> StartStreamingSession(string language) |
|
ModelMetadata<T> GetModelMetadata() |
|
Task<TranscriptionResult<T>> TranscribeAsync(Tensor<T> audio, string language, bool? includeTimestamps, CancellationToken cancellationToken) |
|
Tensor<T> ForwardForTraining(Tensor<T> input) |
|
TranscriptionResult<T> Transcribe(Tensor<T> audio, string language, bool? includeTimestamps) |
Transcribes audio: the LLM’s greedy answer to the transcription prompt, at most MaxTextLength tokens. |
string DetectLanguage(Tensor<T> audio) |