create model card

README.md

@@ -7,4 +7,58 @@ language:
 - en
 - zh
 library_name: transformers
----
+---
+
+**Model type:**
+TinyLLaVA, a tiny model (1.4B) trained using the exact recipe of [llava](https://github.com/haotian-liu/LLaVA).
+We trained our TinyLLaVA using [TinyLlama](https://huggingface.co/PY007/TinyLlama-1.1B-Chat-v0.3) as our LLM backbone, and [clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336) as our vision backbone.
+
+**Model use:**
+The weights have been converted to hf format.
+
+## How to use the model
+
+First, make sure to have `transformers >= 4.35.3`.
+The model supports multi-image and multi-prompt generation. Meaning that you can pass multiple images in your prompt. Make sure also to follow the correct prompt template (`USER: xxx\nASSISTANT:`) and add the token `<image>` to the location where you want to query images:
+
+### Using `pipeline`:
+
+Below we used [`"bczhou/tiny-llava-v1-hf"`](https://huggingface.co/bczhou/tiny-llava-v1-hf) checkpoint.
+
+```python
+from transformers import pipeline
+from PIL import Image
+import requests
+model_id = "bczhou/tiny-llava-v1-hf"
+pipe = pipeline("image-to-text", model=model_id)
+url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
+image = Image.open(requests.get(url, stream=True).raw)
+prompt = "USER: <image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT:"
+outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200})
+print(outputs)
+>>> {"generated_text": "\nUSER: What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT: Lava"}
+```
+
+### Using pure `transformers`:
+
+Below is an example script to run generation in `float16` precision on a GPU device:
+
+```python
+import requests
+from PIL import Image
+import torch
+from transformers import AutoProcessor, LlavaForConditionalGeneration
+model_id = "bczhou/tiny-llava-v1-hf"
+prompt = "USER: <image>\nWhat are these?\nASSISTANT:"
+image_file = "http://images.cocodataset.org/val2017/000000039769.jpg"
+model = LlavaForConditionalGeneration.from_pretrained(
+model_id,
+torch_dtype=torch.float16,
+low_cpu_mem_usage=True,
+).to(0)
+processor = AutoProcessor.from_pretrained(model_id)
+raw_image = Image.open(requests.get(image_file, stream=True).raw)
+inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16)
+output = model.generate(**inputs, max_new_tokens=200, do_sample=False)
+print(processor.decode(output[0][2:], skip_special_tokens=True))
+```