Update README.md

README.md

@@ -1,3 +1,94 @@
----
-license: apache-2.0
----
+---
+base_model:
+- meta-llama/Llama-3.1-8B-Instruct
+- google/siglip-so400m-patch14-384
+tags:
+- captioning
+---
+# Model Card for Llama JoyCaption Alpha Two
+
+[Github](https://github.com/fpgaminer/joycaption)
+
+JoyCaption is an image captioning Visual Language Model (VLM) being built from the ground up as a free, open, and uncensored model for the community to use in training Diffusion models.
+
+Key Features:
+- **Free and Open**: It will be released for free, open weights, no restrictions, and just like [bigASP](https://www.reddit.com/r/StableDiffusion/comments/1dbasvx/the_gory_details_of_finetuning_sdxl_for_30m/), will come with training scripts and lots of juicy details on how it gets built.
+- **Uncensored**: Equal coverage of SFW and NSFW concepts. No "cylindrical shaped object with a white substance coming out on it" here.
+- **Diversity**: All are welcome here. Do you like digital art? Photoreal? Anime? Furry? JoyCaption is for everyone. Pains are being taken to ensure broad coverage of image styles, content, ethnicity, gender, orientation, etc.
+- **Minimal Filtering**: JoyCaption is trained on large swathes of images so that it can understand almost all aspects of our world. almost. Illegal content will never be tolerated in JoyCaption's training.
+
+
+## Motivation
+
+Automated descriptive captions enable the training and finetuning of diffusion models on a wider range of images, since trainers are no longer required to either find images with already associated text or write the descriptions themselves. They also improve the quality of generations produced by Text-to-Image models trained on them (ref: DALL-E 3 paper). But to-date, the community has been stuck with ChatGPT, which is expensive and heavily censored; or alternative models, like CogVLM, which are weaker than ChatGPT and have abysmal performance outside of the SFW domain.
+
+I'm building JoyCaption to help fill this gap by performing near or on-par with GPT4o in captioning images, while being free, unrestricted, and open.
+
+
+## How to Get Started with the Model
+
+Please see the [Github](https://github.com/fpgaminer/joycaption) for more details.
+
+Example usage:
+
+```
+import torch
+import torch.amp
+import torchvision.transforms.functional as TVF
+from PIL import Image
+from transformers import AutoTokenizer, LlavaForConditionalGeneration
+IMAGE_PATH = "image.jpg"
+PROMPT = "Write a long descriptive caption for this image in a formal tone."
+MODEL_NAME = "fancyfeast/llama-joycaption-alpha-two-hf-llava"
+# Load JoyCaption
+# bfloat16 is the native dtype of the LLM used in JoyCaption (Llama 3.1)
+# device_map=0 loads the model into the first GPU
+tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME, use_fast=True)
+llava_model = LlavaForConditionalGeneration.from_pretrained(MODEL_NAME, torch_dtype="bfloat16", device_map=0)
+llava_model.eval()
+with torch.no_grad():
+	# Load and preprocess image
+	# Normally you would use the Processor here, but the image module's processor
+	# has some buggy behavior and a simple resize in Pillow yields higher quality results
+	image = Image.open(IMAGE_PATH)
+	if image.size != (384, 384):
+		image = image.resize((384, 384), Image.LANCZOS)
+	image = image.convert("RGB")
+	pixel_values = TVF.pil_to_tensor(image)
+	# Normalize the image
+	pixel_values = pixel_values / 255.0
+	pixel_values = TVF.normalize(pixel_values, [0.5], [0.5])
+	pixel_values = pixel_values.to(torch.bfloat16).unsqueeze(0)
+	# Build the conversation
+	convo = [
+		{
+			"role": "system",
+			"content": "You are a helpful image captioner.",
+		},
+		{
+			"role": "user",
+			"content": PROMPT,
+		},
+	]
+	# Format the conversation
+	convo_string = tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=True)
+	# Tokenize the conversation
+	convo_tokens = tokenizer.encode(convo_string, add_special_tokens=False, truncation=False)
+	# Repeat the image tokens
+	input_tokens = []
+	for token in convo_tokens:
+		if token == llava_model.config.image_token_index:
+			input_tokens.extend([llava_model.config.image_token_index] * llava_model.config.image_seq_length)
+		else:
+			input_tokens.append(token)
+	input_ids = torch.tensor(input_tokens, dtype=torch.long).unsqueeze(0)
+	attention_mask = torch.ones_like(input_ids)
+	# Generate the caption
+	generate_ids = llava_model.generate(input_ids=input_ids.to('cuda'), pixel_values=pixel_values.to('cuda'), attention_mask=attention_mask.to('cuda'), max_new_tokens=300, do_sample=True, suppress_tokens=None, use_cache=True)[0]
+	# Trim off the prompt
+	generate_ids = generate_ids[input_ids.shape[1]:]
+	# Decode the caption
+	caption = tokenizer.decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)
+	caption = caption.strip()
+	print(caption)
+```