bytedance /LatentSync

About LatentSync

LatentSync re-syncs a talking-face video to a new audio track. ByteDance's open-source lip-sync model uses diffusion (the method behind Stable Diffusion). It ships with a Gradio app, a command-line script, training code and eval tools. Inference needs 8-18 GB of VRAM, and you download the checkpoints yourself.

Best for: Researchers and tinkerers who want to dub talking-head or anime clips with a diffusion lip-sync model.

• Re-syncs lips to new audio without a separate motion step
• Turns audio into embeddings with Whisper, then feeds them to the model
• Ships a Gradio web app and a command-line inference script
• Cleans training data: scene cuts, face alignment, quality filters
• Trains the model and SyncNet with configs for different VRAM sizes
• Checks sync confidence and SyncNet accuracy on your clips

1. 1
   Run source setup_env.sh to install packages and download the checkpoints.
2. 2
   Start the Gradio demo with python gradio_app.py.
3. 3
   Or run command-line inference with ./inference.sh.
4. 4
   Raise inference_steps (20-50) for sharper video, or guidance_scale (1.0-3.0) for tighter lip-sync.
5. 5
   To train, process your data with ./data_processing_pipeline.sh, then run ./train_unet.sh or ./train_syncnet.sh.