hkchengrex /MMAudio

About MMAudio

MMAudio makes sound for a video clip, or from a text prompt alone. It lines the audio up with what's happening on screen. You'll need a GPU with about 6 GB of memory. The pretrained models are non-commercial, so check before you sell anything.

Best for: Sound designers who want automatic sound effects or ambience for short clips.

• Makes sound from a video, or from text alone when you skip the video
• Lines up the generated audio with the video frames
• Saves .flac audio and .mp4 video into ./output
• Runs a Gradio web UI for video, text and image input
• Fits in about 6 GB of GPU memory in 16-bit mode
• Ships training, batch evaluation and onset evaluation scripts

1. 1
   Install PyTorch 2.5.1+ with your CUDA build: the command below

   pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118 --upgrade

   Copy
2. 2
   Clone the repo: the command below

   git clone https://github.com/hkchengrex/MMAudio.git

   Copy
3. 3
   Install the package after PyTorch: cd MMAudio then pip install -e .
4. 4
   Run the demo: the command below. Leave out --video for text-to-audio.

   python demo.py --duration=8 --video= --prompt "your prompt"

   Copy
5. 5
   Or start the Gradio app: python gradio_demo.py (default port 7860, set with --port)