Update README.md

README.md

@@ -1,3 +1,115 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+datasets:
+- lmms-lab/llava-critic-113k
+base_model:
+- lmms-lab/llava-onevision-qwen2-72b-ov-sft
+tags:
+- multimodal
+---
+
+# LLaVA-Critic-72B
+
+## Model Summary
+
+`llava-critic-72b` is the first open-source large multimodal model (LMM) designed as a generalist evaluator for assessing model performance across diverse multimodal scenarios. Built on the foundation of `llava-onevision-72b-ov`, it has been finetuned on [LLaVA-Critic-113k](https://huggingface.co/datasets/lmms-lab/llava-critic-113k) dataset to develop its "critic" capacities.
+
+LLaVA-Critic excels in two primary scenarios:
+- 1️⃣ LMM-as-a-Judge: It delivers judgments closely aligned with human, and provides concrete, image-grounded reasons. An open-source alternative to GPT for evaluations.
+- 2️⃣ Preference Learning: Reliable reward signals power up visual chat, leading to LLaVA-OV-Chat [7B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-7b-ov-chat)/[72B](https://huggingface.co/lmms-lab/llava-onevision-qwen2-72b-ov-chat).
+
+As shown in our paper, `llava-critic-72b` matches or even surpasses GPT-4V in providing human-aligned judgments across different evaluation scenarios.
+
+For further details, please refer to the following resources:
+- 📰 Paper: https://arxiv.org/abs/2410.02712
+- 🪐 Project Page: https://llava-vl.github.io/blog/2024-10-03-llava-critic/
+- 📦 Datasets: https://huggingface.co/datasets/lmms-lab/llava-critic-113k
+- 🤗 Model Collections: https://huggingface.co/collections/lmms-lab/llava-critic-66fe3ef8c6e586d8435b4af8
+- 👋 Point of Contact: [Tianyi Xiong](https://tyxiong23.github.io/)
+
+
+## Use
+
+### Intended Use
+
+The model demonstrates general capacities in providing quantitative judgments and qualitative justifications for evaluating LMM-generated responses. It mainly focuses on two evaluation settings:
+- *Pointwise scoring*, where it assigns a score to an individual candidate response.
+- *Pairwise ranking*, where it compares two candidate responses to determine their relative quality.
+
+### Quick Start
+
+~~~python
+# pip install git+https://github.com/LLaVA-VL/LLaVA-NeXT.git
+from llava.model.builder import load_pretrained_model
+from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
+from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN, DEFAULT_IM_START_TOKEN, DEFAULT_IM_END_TOKEN, IGNORE_INDEX
+from llava.conversation import conv_templates, SeparatorStyle
+
+from PIL import Image
+import requests
+import copy
+import torch
+
+import sys
+import warnings
+import os
+
+
+warnings.filterwarnings("ignore")
+pretrained = "lmms-lab/llava-critic-72b"
+model_name = "llava_qwen"
+device = "cuda"
+device_map = "auto"
+tokenizer, model, image_processor, max_length = load_pretrained_model(pretrained, None, model_name, device_map=device_map)  # Add any other thing you want to pass in llava_model_args
+
+model.eval()
+
+url = "https://github.com/LLaVA-VL/blog/blob/main/2024-10-03-llava-critic/static/images/critic_img_seven.png?raw=True"
+image = Image.open(requests.get(url, stream=True).raw)
+image_tensor = process_images([image], image_processor, model.config)
+image_tensor = [_image.to(dtype=torch.float16, device=device) for _image in image_tensor]
+
+conv_template = "qwen_1_5"  # Make sure you use correct chat template for different models
+
+# pairwise ranking
+critic_prompt = "Given an image and a corresponding question, please serve as an unbiased and fair judge to evaluate the quality of the answers provided by a Large Multimodal Model (LMM). Determine which answer is better and explain your reasoning with specific details. Your task is provided as follows:\nQuestion: [What this image presents?]\nThe first response: [The image is a black and white sketch of a line that appears to be in the shape of a cross. The line is a simple and straightforward representation of the cross shape, with two straight lines intersecting at a point.]\nThe second response: [This is a handwritten number seven.]\nASSISTANT:\n"
+
+# pointwise scoring
+# critic_prompt = "Given an image and a corresponding question, please serve as an unbiased and fair judge to evaluate the quality of answer answers provided by a Large Multimodal Model (LMM). Score the response out of 100 and explain your reasoning with specific details. Your task is provided as follows:\nQuestion: [What this image presents?]\nThe LMM response: [This is a handwritten number seven.]\nASSISTANT:\n "
+
+question = DEFAULT_IMAGE_TOKEN + "\n" + critic_prompt
+conv = copy.deepcopy(conv_templates[conv_template])
+conv.append_message(conv.roles[0], question)
+conv.append_message(conv.roles[1], None)
+prompt_question = conv.get_prompt()
+
+input_ids = tokenizer_image_token(prompt_question, tokenizer, IMAGE_TOKEN_INDEX, return_tensors="pt").unsqueeze(0).to(device)
+image_sizes = [image.size]
+
+
+cont = model.generate(
+    input_ids,
+    images=image_tensor,
+    image_sizes=image_sizes,
+    do_sample=False,
+    temperature=0,
+    max_new_tokens=4096,
+)
+text_outputs = tokenizer.batch_decode(cont, skip_special_tokens=True)
+print(text_outputs[0])
+~~~
+
+
+## Citation
+
+```
+@article{xiong2024llavacritic,
+  title={LLaVA-Critic: Learning to Evaluate Multimodal Models},
+  author={Xiong, Tianyi and Wang, Xiyao and Guo, Dong and Ye, Qinghao and Fan, Haoqi and Gu, Quanquan and Huang, Heng and Li, Chunyuan},
+  year={2024},
+  eprint={2410.02712},
+  archivePrefix={arXiv},
+  primaryClass={cs.CV},
+  url={https://arxiv.org/abs/2410.02712},
+}
+```