StepAudio<T>
Models & Types · AiDotNet.TextToSpeech.MultiModal
Step-Audio (Huang et al. 2025): speech understanding and generation for voice interaction. A dual-codebook tokenizer turns speech into linguistic and semantic tokens, the 130B Step-1 chat model answers in text (AQTA), and the 3B Step-1 TTS model writes the answer’s dual-codebook tokens, which CosyVoice’s flow and HiFT turn into 22.05 kHz speech.
Inheritance: NeuralNetworkBase<T> → TtsModelBase<T> → StepAudio<T>
For Beginners
Step-Audio listens, thinks in text with a very large language model, and answers aloud in a voice you give it a sample of; tags like “(高兴1)” or “(RAP)” at the start of the text set the style.
How It Works
Built as the release (stepfun-ai/Step-Audio) runs it; see StepAudioOptions and
docs/text-to-speech/paper-fidelity-decisions.md. Speech goes in at any rate (resampled to 16 kHz for the
tokenizer) and comes out at 22.05 kHz. The TTS speaks as SpeakerVoice, a speaker made from a prompt
recording and its transcript (CreateSpeaker).
Constructors
| Constructor | Summary |
|---|---|
StepAudio(NeuralNetworkArchitecture<T> architecture) |
Calls the fully specified overload with the omitted parameters’ defaults. |
StepAudio(NeuralNetworkArchitecture<T> architecture, StepAudioOptions options) |
Creates Step-Audio with its networks (randomly initialized). |
StepAudio(NeuralNetworkArchitecture<T> architecture, string modelPath, StepAudioOptions options) |
Creates Step-Audio running an ONNX export. |
Properties
| Property | Summary |
|---|---|
StepAudioSpeaker<T> SpeakerVoice { get; set; } |
The speaker the TTS speaks as; Synthesize needs one. |
bool TrainsOnSampleObjective { get; } |
|
int MaxTextLength { get; } |
The longest answer, in tokens. |
Methods
| Method | Summary |
|---|---|
LayerInputDomain GetInputDomain(int[] inputShape) |
|
ModelMetadata<T> GetModelMetadata() |
|
StepAudioSpeaker<T> CreateSpeaker(Tensor<T> recording, int sampleRate, string transcript, string name) |
A speaker from a prompt recording (any rate) and its transcript, named as the release’s system prompt names speakers (empty for the prompt without a name). |
T EvaluateTrainingObjective(TtsTrainingSample<T> sample) |
|
T TrainChat(Tensor<T> question, int sampleRate, string answer, IReadOnlyList<ValueTuple<string, string>> history) |
One AQTA fine-tuning step (§5.2.2) on the chat model: the chat prompt with earlier turns and the spoken question, then the answer’s tokens and <|EOT|>, which alone are scored. |
T TrainContinuation(Tensor<T> recording, int sampleRate, bool tts) |
A pre-training step on a recording’s audio continuation (its dual-codebook tokens). |
T TrainPpo(Tensor<T> question, int sampleRate) |
One PPO rollout (§5.2.6) on a spoken question: the policy (the chat model) samples an answer; the per-token reward is −β (log π − log π_ref) with the reward model’s score added at the last token; GAE (γ, λ) gives the advantages (whitened) a… |
T TrainReward(Tensor<T> question, int sampleRate, string chosen, string rejected) |
One reward-model step (§5.2.4): the Bradley–Terry loss −log σ(r(chosen) − r(rejected)) on two answers to the same spoken question. |
T TrainSequence(string sequence, bool tts) |
One pre-training step (§4.2) on a sequence of text and <audio_N> tokens (pure audio continuation: AudioTokenString; interleaved, ASR and TTS data: the caller’s layout, whose format the paper does not state), on the chat model or, with `tt… |
T TrainTts(string text, Tensor<T> recording, int sampleRate, StepAudioSpeaker<T> speaker) |
One TTS fine-tuning step (§5.1.2) on the TTS model: the two-turn prompt with the speaker (their transcript and audio tokens) and the human turn’s text, then the recording’s dual-codebook tokens and <|EOT|>, which alone are scored. |
Tensor<T> Speak(string text) |
Speech for text in SpeakerVoice (the release’s TTS), 22.05 kHz. |
Tensor<T> Synthesize(string text) |
|
ValueTuple<Tensor<T>, string> Chat(Tensor<T> speech, int sampleRate) |
The full voice interaction (stepaudio.py’s __call__): the spoken answer to a spoken question, and its text. |
double Reward(Tensor<T> question, int sampleRate, string answer) |
The reward model’s score of an answer to a spoken question (its value at the last token). |
int[] TokenizeText(string text) |
Step-1’s token ids of text (with |
string Answer(Tensor<T> speech, int sampleRate, IReadOnlyList<ValueTuple<string, string>> history) |
The chat model’s text answer to a spoken question (AQTA), the question at sampleRate, after optional earlier turns (role, text). |
string Answer(string question) |
The chat model’s text answer to a text question (TQTA). |
string AudioTokenString(Tensor<T> recording, int sampleRate) |
The dual-codebook token string of a recording (“<audio_N>…”, tokenizer.py’s __call__). |
void InitializeCriticFromReward() |
Starts the critic from the trained reward model. |
void InitializeRlhfFromChat() |
Starts the reward model from the current chat model (§5.2.4: “initializes from a SFT model”) and the critic and the frozen reference policy likewise. |