KlingAIResearch /X-Dub

About X-Dub

X-Dub takes a video plus an audio file and redraws the mouth to match. It's the official code for a research paper, and the public build runs on Wan2.2-TI2V-5B. Budget about 21 GB of VRAM, and it handles one person per clip. The public release is weaker than the paper version: flicker, drift (the face slowly changing) and noisy frames.

Best for: Re-dub artists who need to match one on-screen character's mouth to a new audio track.

• Dubs one on-screen person's mouth to a new audio track
• Skips masking: no manual mouth-region mask needed
• Auto-crops any video size, then pastes the result back
• Uses DWPose (a pose tool that finds face points) to place the crop
• Lets you tune ref_cfg_scale, audio_cfg_scale and 25-50 steps
• Runs inside ComfyUI through community-made nodes

1. 1
   Clone the repo and make a Python 3.10 env: the command below, the next command, conda activate x-dub

   git clone https://github.com/KlingAIResearch/X-Dub.git

   Copy

   conda create -n x-dub python=3.10 -y

   Copy
2. 2
   Install dependencies: the command below, then the OpenMMLab packages (mmengine, mmcv, mmdet, mmpose, chumpy), then pip install -e . --no-deps

   pip install -r requirements.txt

   Copy
3. 3
   Download the weights into checkpoints/: the command below

   hf download KlingTeam/X-Dub --local-dir ./checkpoints --repo-type model

   Copy
4. 4
   Move the DWPose files: mkdir -p dwpose_tools/models then the command below

   cp -r ./checkpoints/dwpose_tools/models/. ./dwpose_tools/models/

   Copy
5. 5
   Run inference: the command below

   python infer_lip_sync_pipeline.py --video_path assets/examples/video.mp4 --audio_path assets/examples/audio.wav --ckpt_path checkpoints/X-Dub_model.safetensors --ref_cfg_scale 2.5 --audio_cfg_scale 10.0 --num_inference_steps 30 --output_dir ./results

   Copy