Dia<T>
Models & Types · AiDotNet.TextToSpeech.CodecBased
Dia, Nari Labs’ dialogue model (2025): a Transformer encoder reads the script’s UTF-8 bytes — speaker tags [S1] and [S2] as bytes 1 and 2, non-verbal cues such as “(laughs)” as plain text — and an encoder-decoder Transformer predicts DAC’s nine codebooks at 86 frames a second, the codebooks delayed against each other, under classifier-free guidance; DAC decodes the codes to 44.1 kHz audio.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → Dia<T>
Implements: ICodecTts<T>
For Beginners
Dia speaks a written dialogue. Mark who speaks with [S1] and [S2], add cues like “(laughs)”, and it voices both speakers in one recording. Give it a short recording and its transcript to make it speak in that voice.
How It Works
Nari Labs released code and weights, not a paper (nari-labs/dia dia/layers.py, dia/model.py,
dia/audio.py; nari-labs/Dia-1.6B-0626). The decoder reads, at each step, the sum of the nine channels’
embeddings and predicts all nine; channel c runs delay[c] steps behind channel 0 ([0, 8, …, 15]), so a step’s
codes are drawn knowing the earlier codebooks of the same frame. Generation runs the decoder twice per step, on the
script and on an all-zero script, keeps the codes whose guided logits cond + s · (cond − uncond) are among the
best k, and samples from the conditional logits among them with temperature and top-p; channel 0’s end token starts a
countdown that closes each delayed channel with the end token and pads after it. An audio prompt’s codes, with its
transcript at the front of the script, are prefilled to clone its voice.
Training follows transformers’ DiaForConditionalGeneration, written with Nari Labs and the only training code: the decoder reads the delayed codes between start and end tokens, and every channel’s next token is scored by cross-entropy, padding and start tokens excluded; the optimizer is the Trainer’s default AdamW.
Example
var architecture = new NeuralNetworkArchitecture<float>(InputType.OneDimensional, NeuralNetworkTaskType.Regression, inputSize: 1, outputSize: 1);
var model = new Dia<float>(architecture, new DiaOptions());
model.LoadReferenceWeights("Dia-1.6B-0626/dia-v1.pth");
model.Codec.LoadDescriptCheckpoint("weights_44khz_8kbps_0.0.1.pth");
var audio = model.Synthesize("[S1] Dia is an open weights text to dialogue model. [S2] Wow. Amazing. (laughs)");
Constructors
| Constructor | Summary |
|---|---|
Dia(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
Dia(NeuralNetworkArchitecture<T> architecture, DiaOptions options) |
Creates a native, trainable Dia. |
Dia(NeuralNetworkArchitecture<T> architecture, string modelPath, DiaOptions options) |
Creates an ONNX-backed Dia for inference. |
Properties
| Property | Summary |
|---|---|
DAC<T> Codec { get; } |
The DAC codec whose codes Dia predicts (load its released weights with LoadDescriptCheckpoint). |
bool TrainsOnSampleObjective { get; } |
|
int CodebookSize { get; } |
|
int CodecFrameRate { get; } |
|
int CodecTokenCodebooks { get; } |
|
int CodecTokenVocabulary { get; } |
|
int MaxTextLength { get; } |
|
int NumCodebooks { get; } |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
Tensor<T> DecodeFromTokens(Tensor<T> tokens) |
|
Tensor<T> EncodeToTokens(Tensor<T> audio) |
|
Tensor<T> Generate(string script, int[] audioPrompt) |
Speaks script (Dia.generate), continuing audioPrompt’s codes [frames, 9] when given (put the prompt’s transcript at the front of the script); returns only the new audio. |
Tensor<T> Synthesize(string text) |
|
int[] Tokenize(string text, int maxLength) |
The script’s byte ids (Dia._encode_text): its UTF-8 bytes with “[S1]” and “[S2]” as bytes 1 and 2, cut to MaxTextLength. |
void LoadReferenceWeights(string path) |
Loads Dia’s weights (nari-labs/Dia-1.6B-0626): the release’s dia-v1.pth (DiaModel‘s names and DenseGeneral kernels), transformers’ model.safetensors (one file, or the shards its model.safetensors.index.json lists), or a director… |