JEN1Model<T>
Models & Types · AiDotNet.Diffusion.Audio
JEN-1 model for high-fidelity text-to-music generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → AudioDiffusionModelBase<T> → JEN1Model<T>
For Beginners
JEN-1 creates music from text descriptions:
How JEN-1 works:
- Text is encoded by a FLAN-T5 text encoder
- Audio is encoded by a 1D VAE into a compressed latent representation
- A diffusion model denoises latents conditioned on text
- The 1D VAE decodes latents back to 48kHz audio
Key characteristics:
- High-fidelity 48kHz audio output
- Text-to-music: Generate music from text descriptions
- Music continuation: Extend an existing music clip
- Music inpainting: Fill in missing parts of music
- Multi-task training: Autoregressive + non-autoregressive
Advantages:
- High audio quality (48kHz)
- Versatile (text-to-music, continuation, inpainting)
- Good musicality and coherence
- Efficient 1D latent representation
Limitations:
- Limited to ~10 second clips
- Generation takes several seconds
- Music quality varies with prompt complexity
How It Works
JEN-1 is a universal high-fidelity music generation model that combines autoregressive and non-autoregressive training in a multi-task framework. It generates music at 48kHz in both text-to-music and music continuation modes.
Technical specifications:
- Architecture: 1D VAE + Latent Diffusion
- Audio encoder: 1D convolutional VAE, 128 latent channels
- Diffusion backbone: 1D U-Net with cross-attention
- Text encoder: FLAN-T5 Large (1024-dim embeddings)
- Sample rate: 48,000 Hz
- Duration: Up to 10 seconds
- Noise schedule: Linear beta, 1000 timesteps
Reference: Li et al., “JEN-1: Text-Guided Universal Music Generation with Omnidirectional Diffusion Models”, 2023
Example
var jen1 = new JEN1Model<float>();
var audio = jen1.GenerateFromText(
prompt: "Upbeat electronic dance music with a driving bass line",
durationSeconds: 10.0,
numInferenceSteps: 100,
guidanceScale: 3.0,
seed: 42);
Constructors
| Constructor | Summary |
|---|---|
JEN1Model(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, AudioVAE<T> audioVae, IConditioningModule<T> conditioner, int? seed) |
Initializes a new instance of JEN1Model with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsAudioToAudio { get; } |
|
bool SupportsTextToAudio { get; } |
|
bool SupportsTextToMusic { get; } |
|
bool SupportsTextToSpeech { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
ModelMetadata<T> GetModelMetadata() |