MeiGen-AI /InfiniteTalk

About InfiniteTalk

Give InfiniteTalk a video or still photo plus audio, and it makes a new video where mouth, head and body follow the sound. Streaming mode keeps going past one minute for long dubs, though you'll need a big GPU and some setup. After about a minute, colors drift and the face matches less well.

Best for: Dubbing editors who want to re-animate existing footage or a portrait photo with matching lip sync.

• Syncs lips, head, body and expressions to your audio track
• Dubs a video, or animates one photo with an audio track
• Streams unlimited length; clip mode makes one short chunk
• Exports 480P or 720P video
• Speeds up with TeaCache, int8 quantization and FusionX or lightx2v LoRAs
• Runs on several GPUs, or on very low VRAM with a flag

1. 1
   Make a conda env named multitalk and install PyTorch 2.4.1 and xformers 0.0.28 with CUDA 12.1 wheels.
2. 2
   Install flash-attn 2.7.4.post1, then run the command below and the next command.

   pip install -r requirements.txt

   Copy

   conda install -c conda-forge librosa

   Copy
3. 3
   Install ffmpeg, then download the base model, audio encoder and InfiniteTalk weights with huggingface-cli download.
4. 4
   Run single-GPU inference: the command below

   python generate_infinitetalk.py --ckpt_dir weights/Wan2.1-I2V-14B-480P --wav2vec_dir 'weights/chinese-wav2vec2-base' --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors --input_json examples/single_example_image.json --size infinitetalk-480 --sample_steps 40 --mode streaming --motion_frame 9 --save_file infinitetalk_res

   Copy
5. 5
   Or launch the Gradio demo: the command below

   python app.py --ckpt_dir weights/Wan2.1-I2V-14B-480P --wav2vec_dir 'weights/chinese-wav2vec2-base' --infinitetalk_dir weights/InfiniteTalk/single/infinitetalk.safetensors --num_persistent_param_in_dit 0 --motion_frame 9

   Copy