SpiritLM<T>

Models & Types · AiDotNet.TextToSpeech.MultiModal

SpiritLM (Nguyen et al., TACL 2025): an interleaved spoken and written language model. LLaMA 2 7B, continuously pre-trained on text, speech and word-level interleaved speech-text sequences, reads and writes both modalities, each span marked [Text] or [Speech].

Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → SpiritLM<T>

For Beginners

SpiritLM writes text and speech in one stream: it can continue a recording as speech or as text, or read text aloud, because it learned both “languages” at once.

How It Works

Speech tokens (§3.1): the 25 Hz mHuBERT’s layer-11 units, consecutive repeats merged, [Hu0] … [Hu500]. SpiritLM Expressive (§3.2) adds pitch tokens (a VQ-VAE over FCPE’s F0, 12.5 a second) and style tokens (one a second), deduplicated and interleaved with the units by time.

Speech is voiced by a unit HiFi-GAN trained on Expresso with a duration predictor (Base: 4 speakers and 34 styles; Expressive: the pitch and style tokens too).

Training (§3.3, App. A): next-token prediction over text-only ([Text]…), speech-only ([Speech]…) and aligned sequences whose modality switches at word boundaries, text spans of 10–30 words and speech spans of 5–15, at a constant 3e-5 with RoPE’s base at 100,000.

Example

var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new SpiritLM<float>(architecture, new SpiritLMOptions());
model.LoadReleasedWeights("spiritlm_model/spirit-lm-base-7b", "speech_tokenizer");
var speech = model.Synthesize("The largest country in the world is");
var continuation = model.Generate(new[] { SpiritLmContent<float>.FromSpeech(recording16kHz) }, SpiritLmModality.Text);

Constructors

Constructor Summary
SpiritLM(NeuralNetworkArchitecture<T> architecture) Calls the fully specified overload with the omitted parameters’ defaults.
SpiritLM(NeuralNetworkArchitecture<T> architecture, SpiritLMOptions options) Creates a native, trainable SpiritLM.
SpiritLM(NeuralNetworkArchitecture<T> architecture, string modelPath, SpiritLMOptions options) Creates an ONNX-backed SpiritLM for inference.

Properties

Property Summary
EnCodec<T> SpeechPropCodec { get; } SONAR Expressive’s 25 Hz EnCodec (with SpeechProp; otherwise null): load or train it as a codec model of its own.
bool IsExpressive { get; } Whether this is SpiritLM Expressive.
bool TrainsOnSampleObjective { get; }
int MaxTextLength { get; } The longest sequence the language model takes (4096).

Methods

Method Summary
IReadOnlyList<SpiritLmContent<T>> Generate(IReadOnlyList<SpiritLmContent<T>> inputs, SpiritLmModality output) Generates output after interleaved text and speech (the release’s Spiritlm.generate): the prompt built, the other modality’s tokens forbidden unless the output is arbitrary, and the result decoded — text as it is, speech voiced.
IReadOnlyList<SpiritLmContent<T>> Generate(string prompt, SpiritLmModality output, int maxNewTokens) Generates after a prompt already written as a string ([Text]…, [Speech][Hu…]…).
LayerInputDomain GetInputDomain(int[] inputShape)
ModelMetadata<T> GetModelMetadata()
T EvaluateTrainingObjective(TtsTrainingSample<T> sample)
T TrainInterleaved(Tensor<T> recording, IReadOnlyList<SpiritLmWord> words) One step on an aligned recording interleaved at word boundaries (§3.1’s speech+text data).
T TrainPitchQuantizer(Tensor<T> recording, SpiritLmPitchExtractor extractor) One step of SpiritLM Expressive’s F0 VQ-VAE (§3.2) on 16 kHz speech: its F0 by pYAAPT (the paper’s extractor for this model; extractor to change it), interpolated and normalized by the voiced mean, with the voicing, cut to a multiple of 1…
T TrainSequence(string sequence) One continued pre-training step on a sequence (TextSequence, SpeechSequence, InterleavedSequence), tokenized with <s>.
T TrainSonarExpressive(Tensor<T> recording, Tensor<T> semantic, bool withSpeechProp) One step of SONAR Expressive (§4.1) on a recording and the SONAR embedding semantic ([1024], of its transcript or translation): the decoders’ cross-entropy on its EnCodec units.
T TrainSonarExpressive(Tensor<T> recording, string transcript, string language, bool withSpeechProp) The step of TrainSonarExpressive with the semantic embedding of transcript in language (e.g.
T TrainSpeech(Tensor<T> recording) One step on speech alone (§3.1’s speech-only data).
T TrainSpeechPropStyle(Tensor<T> recording, int style) One step of the style fine-tuning (§3.2): the SpeechProp features of each second of a recording classified into Expresso’s styles by a linear head, cross-entropy against style; SpeechProp’s encoder trains with the head.
T TrainText(string text) One step on text alone (§3.1’s text-only data).
T TrainVocoder(Tensor<T> recording) Calls the fully specified overload with the omitted parameters’ defaults.
T TrainVocoder(Tensor<T> recording, int speaker) Calls the fully specified overload with the omitted parameters’ defaults.
T TrainVocoder(Tensor<T> recording, int speaker, int? style) One GAN step of the unit vocoder (speech-resynthesis’ and Expresso’s train.py) on 16 kHz speech of a speaker: its frame-level units (with the per-frame pitch and style tokens for Expressive, or the utterance style for Base) and audio cu…
T TrainVocoder(Tensor<T> recording, int? style) Calls the fully specified overload with the omitted parameters’ defaults.
T TrainVocoder(Tensor<T> recording, int[] codes, int[] pitch, int[] styles, int speaker, int? style) The vocoder step on a recording whose speech tokens are already extracted (SpeechUnits), as the release’s training reads them from a manifest: frame-level HuBERT units and, for Expressive, the pitch and style tokens at their own rates.
Tensor<T> SonarEmbedding(string text, string language) SONAR’s sentence embedding [1024] of text.
Tensor<T> SonarExpressiveSpeech(Tensor<T> semantic, Tensor<T> styleReference, int maxFrames) Speech of a SONAR embedding and a SpeechProp embedding (from styleReference, or none): SONAR Expressive’s units (top-k 10, guidance 3), decoded by its EnCodec.
Tensor<T> Speak(string speechTokens, int? speaker) The speech of a token string (the release’s decode): its HuBERT units, with the pitch and style each carries, voiced with duration prediction as speaker (Speaker by default) and, for Base, VocoderStyle.
Tensor<T> SpeechPropStyleFeatures(Tensor<T> recording) The SpeechProp style features of 16 kHz audio, without gradients.
Tensor<T> Synthesize(string text) The speech SpiritLM writes after [Text]{text}[Speech] (text-to-speech continuation, App.
ValueTuple<int[], int[], int[]> SpeechUnits(Tensor<T> recording) The speech tokens of 16 kHz audio (the release’s encode_units): per chunk of at most 100 seconds (chunks under 1280 samples skipped), the HuBERT units and, for Expressive, the pitch and style tokens.
int[] Encode(string text, bool addBos) The ids of a string as the release’s LlamaTokenizer (legacy, add_bos_token) encodes it.
int[] TokenizeText(string text) The token ids of text (no <s>).
string Decode(IEnumerable<int> ids) The string of ids (<s> and </s> skipped).
string InterleavedSequence(Tensor<T> recording, IReadOnlyList<SpiritLmWord> words, Random random) An interleaved training sequence (§3.1, App.
string Prompt(IReadOnlyList<SpiritLmContent<T>> inputs, SpiritLmModality output) The prompt of interleaved inputs (the release’s _build_prompt): each run of a modality opened by its marker, speech as its token string, and the output modality’s marker when the last input differs.
string SpeechSequence(Tensor<T> recording) A speech-only training sequence (§3.1): [Speech]{speech tokens}.
string SpeechString(Tensor<T> recording) The speech-token string of 16 kHz audio (the release’s encode_string).
string TextSequence(string text) A text-only training sequence (§3.1): [Text]{text}.
void FitStyleCentroids(IEnumerable<Tensor<T>> recordings, int iterations) Fits the style k-means (§3.2: 100 clusters of the normalized features) on the SpeechProp style features of recordings: k-means++ seeding, then Lloyd’s iterations until no assignment changes or iterations have run.
void LoadFcpe(string path) Loads FCPE’s weights, torchfcpe’s bundled assets/fcpe_c_v001.pt (SpiritLM Expressive with Fcpe).
void LoadReleasedWeights(string modelDirectory, string speechTokenizerDirectory) Loads a released SpiritLM: the language model of modelDirectory (spirit-lm-base-7b or spirit-lm-expressive-7b; its tokenizer.model unless TokenizerPath is set) and, with speechTokenizerDirectory, the release’s speech_tokenizer…
void LoadSonarTextEncoder(string directory) Loads SONAR’s text encoder (transformers’ M2M100Encoder, e.g.
void LoadSpeechTokenizer(string directory) Loads the release’s speech_tokenizer directory: the 25 Hz mHuBERT and its linear quantizer (hubert_25hz), the variant’s vocoder (hifigan_spiritlm_base or hifigan_spiritlm_expressive_w2v2) and, for Expressive, the F0 VQ-VAE (`vqvae_f…
void LoadUnitKMeans(string path) Loads the k-means centroids of KMeans: a joblib-pickled scikit-learn model (as TWIST’s and textlesslib’s tokenizers ship them) or a torch file with cluster_centers (the release’s KmeansModel format).