Description
Cross-modal generative media connecting audio and visual neural networks. Students explore spectrogram diffusion, neural voice synthesis, and multi-modal synchronization.
Key Topics
- Spectrogram Diffusion: Treating sound frequencies as 2D image spectrograms to generate audio with image diffusion models (Riffusion).
- Text-to-Audio / Music Models: AudioCraft, MusicGen, and latent audio embeddings.
- Audio-Reactive Latent Navigation: Modulating diffusion seeds and camera trajectories using audio RMS and FFT metrics.