CSM<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
CSM, Sesame’s Conversational Speech Model (Sesame 2025, “Crossing the uncanny valley of conversational voice”): a Llama backbone reads the conversation — text and Mimi’s audio codes interleaved, one frame per position — and predicts each next audio frame’s first codebook; a small Llama audio decoder then predicts the frame’s other 31 codebooks one by one from the backbone’s state; Mimi decodes the frames to 24 kHz audio, which SilentCipher watermarks.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → CSM<T>
Implements: ICodecTts<T>
For Beginners
CSM speaks one turn of a conversation at a time. Give it the earlier turns (who said what, and how it sounded) and it says the next line in a voice and manner that fit them.
How It Works
Sesame published a blog post, not a paper, and released inference code (SesameAILabs/csm models.py,
generator.py, watermarking.py). The blog fixes the design — two autoregressive Transformers over Mimi’s
split-RVQ tokens (one semantic and 31 acoustic codebooks at 12.5 Hz), a Llama tokenizer for text, the zeroth codebook
predicted by the backbone and the rest by the decoder with a separate head for each — and its training’s compute
amortization: the decoder learns on a random 1/16 of the frames, the zeroth codebook on all of them. The release fills
the rest. Each position of the backbone is a frame of 33 tokens (32 audio codes and one text token) of which a mask
keeps either the text token or the codes; the embeddings it keeps are summed. A turn is its text — “[speaker]words”
between Llama 3’s begin and end of text, one token a frame — then its audio’s frames and a frame of zeros. Generation
samples every codebook with top-k 50 at temperature 0.9 until a frame of zeros, the decoder running afresh from the
backbone’s last state for each frame.
Training follows the blog’s split and Hugging Face’s CSM port (written with Sesame), which is the only training code: the backbone’s next-frame cross-entropy on codebook 0 over the audio frames, plus the decoder’s cross-entropy on codebooks 1–31 over the frames chosen for it, with the Trainer’s default AdamW.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new CSM<float>(architecture, new CSMOptions { WatermarkModelPath = "silentcipher/44_1_khz/73999_iteration" });
model.LoadReferenceWeights("csm-1b/ckpt.pt");
model.LoadCodecWeights("moshiko-pytorch-bf16/tokenizer-e351c8d8-checkpoint125.safetensors");
var audio = model.Generate("Hello from Sesame.", speaker: 0, context: Array.Empty<CsmSegment<float>>());
Constructors
| Constructor | Summary |
|---|---|
CSM(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
CSM(NeuralNetworkArchitecture<T> architecture, CSMOptions options) |
Creates a native, trainable CSM. |
CSM(NeuralNetworkArchitecture<T> architecture, string modelPath, CSMOptions options) |
Creates an ONNX-backed CSM for inference. |
Properties
| Property | Summary |
|---|---|
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Generate(string text, int speaker, IReadOnlyList<CsmSegment<T>> context) |
Speaks text as speaker’s next turn after context (Generator.generate): each earlier turn’s text and audio frames, then this turn’s text, then frames sampled until a frame of zeros or MaxAudioLengthMs, decoded by Mimi and watermark… |
Tensor<T> Synthesize(string text) |
|
void LoadCodecWeights(string path) |
Loads Mimi: kyutai/moshiko-pytorch-bf16’s tokenizer-e351c8d8-checkpoint125.safetensors, as the release does. |
void LoadReferenceWeights(string path) |
Loads the release’s weights (sesame/csm-1b): its ckpt.pt or a .safetensors of the same state dict (models.Model’s names: torchtune’s backbone.* and decoder.*, text_embeddings, audio_embeddings, projection, codebook0_head… |