Add readme and model_index.json

README.md

@@ -1,3 +1,112 @@
----
-license: mit
----
+---
+license: mit
+tags:
+  - stable-diffusion
+  - stable-diffusion-diffusers
+  - image-to-image
+  - art
+widget:
+  - src: >-
+      https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png
+    prompt: Cartoonize the following image
+datasets:
+- instruction-tuning-sd/cartoonization
+---
+
+# Instruction-tuned Stable Diffusion for Cartoonization (Fine-tuned) 
+
+This pipeline is an 'instruction-tuned' version of [Stable Diffusion (v1.5)](https://huggingface.co/runwayml/stable-diffusion-v1-5). It was
+fine-tuned from the existing [InstructPix2Pix checkpoints](https://huggingface.co/timbrooks/instruct-pix2pix).
+
+## Pipeline description
+
+Motivation behind this pipeline partly comes from [FLAN](https://huggingface.co/papers/2109.01652) and partly
+comes from [InstructPix2Pix](https://huggingface.co/papers/2211.09800). The main idea is to first create an
+instruction prompted dataset (as described in [our blog](https://hf.co/blog/instruction-tuning-sd)) and then conduct InstructPix2Pix style
+training. The end objective is to make Stable Diffusion better at following specific instructions
+that entail image transformation related operations.
+
+<p align="center">
+<img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/instruction-tuning-sd.png" width=600/>
+</p>
+
+Follow [this post](https://hf.co/blog/instruction-tuning-sd) to know more. 
+
+## Training procedure and results
+
+Training was conducted on [instruction-tuning-sd/cartoonization](https://huggingface.co/datasets/instruction-tuning-sd/cartoonization) dataset. Refer to
+[this repository](https://github.com/huggingface/instruction-tuned-sd) to know more. The training logs can be found [here](https://wandb.ai/sayakpaul/instruction-tuning-sd?workspace=user-sayakpaul). 
+
+Here are some results dervied from the pipeline:
+
+<p align="center">
+<img src="https://huggingface.co/datasets/sayakpaul/sample-datasets/resolve/main/cartoonization_results.jpeg" width=600/>
+</p>
+
+## Intended uses & limitations
+
+You can use the pipeline for performing cartoonization with an input image and an input prompt.
+
+### How to use
+
+Here is how to use this model:
+
+```python
+import torch
+from diffusers import StableDiffusionInstructPix2PixPipeline
+from diffusers.utils import load_image
+
+model_id = "instruction-tuning-sd/cartoonizer"
+pipeline = StableDiffusionInstructPix2PixPipeline.from_pretrained(
+    model_id, torch_dtype=torch.float16, use_auth_token=True
+).to("cuda")
+
+image_path = "https://hf.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"
+image = load_image(image_path)
+
+image = pipeline("Cartoonize the following image", image=image).images[0]
+image.save("image.png")
+```
+
+For notes on limitations, misuse, malicious use, out-of-scope use, please refer to the model card
+[here](https://huggingface.co/runwayml/stable-diffusion-v1-5).
+
+## Citation
+
+**FLAN**
+
+```bibtex
+@inproceedings{
+    wei2022finetuned,
+    title={Finetuned Language Models are Zero-Shot Learners},
+    author={Jason Wei and Maarten Bosma and Vincent Zhao and Kelvin Guu and Adams Wei Yu and Brian Lester and Nan Du and Andrew M. Dai and Quoc V Le},
+    booktitle={International Conference on Learning Representations},
+    year={2022},
+    url={https://openreview.net/forum?id=gEZrGCozdqR}
+}
+```
+
+**InstructPix2Pix**
+
+```bibtex
+@InProceedings{
+    brooks2022instructpix2pix,
+    author     = {Brooks, Tim and Holynski, Aleksander and Efros, Alexei A.},
+    title      = {InstructPix2Pix: Learning to Follow Image Editing Instructions},
+    booktitle  = {CVPR},
+    year       = {2023},
+}
+```
+
+**Instruction-tuning for Stable Diffusion blog**
+
+```bibtex
+@article{
+  Paul2023instruction-tuning-sd,
+  author = {Paul, Sayak},
+  title = {Instruction-tuning Stable Diffusion with InstructPix2Pix},
+  journal = {Hugging Face Blog},
+  year = {2023},
+  note = {https://huggingface.co/blog/instruction-tuning-sd},
+}
+```

model_index.json

@@ -0,0 +1,33 @@
+{
+  "_class_name": "StableDiffusionInstructPix2PixPipeline",
+  "_diffusers_version": "0.15.0.dev0",
+  "feature_extractor": [
+    "transformers",
+    "CLIPImageProcessor"
+  ],
+  "requires_safety_checker": false,
+  "safety_checker": [
+    "stable_diffusion",
+    "StableDiffusionSafetyChecker"
+  ],
+  "scheduler": [
+    "diffusers",
+    "EulerAncestralDiscreteScheduler"
+  ],
+  "text_encoder": [
+    "transformers",
+    "CLIPTextModel"
+  ],
+  "tokenizer": [
+    "transformers",
+    "CLIPTokenizer"
+  ],
+  "unet": [
+    "diffusers",
+    "UNet2DConditionModel"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKL"
+  ]
+}