HVision-NKU /StoryDiffusion

About StoryDiffusion

StoryDiffusion holds one character steady across a long run of generated images. Give it 3 or more text prompts, and it draws them as a single comic. The project is the official code behind a NeurIPS 2024 paper. The catch: the video model's code and weights were never released.

Best for: Researchers and tinkerers testing character consistency in comics and image-to-video.

• Keeps characters consistent across a long sequence of text prompts
• Plugs into any SD1.5 or SDXL-based image model without retraining
• Needs at least 3 prompts; 5 to 6 gives a better layout
• Predicts motion between condition images to build longer videos
• Ships a Jupyter notebook and a Gradio demo for comics
• Runs a low-VRAM demo tested on a 24GB GPU and 30GB RAM

1. 1
   Make a Python 3.10 environment: the command below

   conda create --name storydiffusion python=3.10

   Copy
2. 2
   Turn it on: the command below

   conda activate storydiffusion

   Copy
3. 3
   Install the requirements: pip install -U pip then the command below

   pip install -r requirements.txt

   Copy
4. 4
   For comics, open Comic_Generation.ipynb and run the cells.
5. 5
   For the local demo, run the command below (low-VRAM version, tested on a 24GB GPU).

   python gradio_app_sdxl_specific_id_low_vram.py

   Copy

• StoryDiffusion vs aimangastudio