Instruction-tuned Diffusion Models

We explore instruction-tuning to teach Stable Diffusion to follow instructions to translate or process input images. With this method, we can prompt our model using an input image and an “instruction”, such as - Apply a cartoon filter to the natural image.

Please follow our blog post for more details: https://huggingface.co/blog/instruction-tuning-sd.

Below we provide links to our training code, models, and datasets:

• Training and inference code

• Models:

  • instruction-tuning-sd/scratch-low-level-img-proc
  • instruction-tuning-sd/scratch-cartoonizer
  • instruction-tuning-sd/cartoonizer
  • instruction-tuning-sd/low-level-img-proc

• Datasets:

  • Instruction-prompted cartoonization
  • Instruction-prompted low-level image processing

Happy instruction-tuning 🤗