update readme

README.md

@@ -4,10 +4,9 @@ language:
 - en
 ---
 
-# RedPajama-Chat-INCITE-2.8B
+# RedPajama-Instruct-INCITE-2.8B
 
-RedPajama-Chat-INCITE-2.8B-v1, is a large transformer-based language model developed by Together Computer and trained on the RedPajama-Data-1T dataset.
-It is further fine-tuned on GPT-JT's datasets enhance zero/few-shot in-context learning.
+RedPajama-Instruct-INCITE-2.8B-v1, is a large transformer-based language model developed by Together Computer and trained on the RedPajama-Data-1T dataset.
 
 ## Model Details
 - **Developed by**: Together Computer.
@@ -18,52 +17,115 @@ It is further fine-tuned on GPT-JT's datasets enhance zero/few-shot in-context l
 
 # Quick Start
 
+Please note that the model requires `transformers` version >= 4.25.1.
+
 ## GPU Inference
 
 This requires a GPU with 8GB memory.
+
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1", torch_dtype=torch.float16)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1", torch_dtype=torch.float16)
 model = model.to('cuda:0')
 # infer
-inputs = tokenizer("Hello", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
 ## GPU Inference in Int8
 
 This requires a GPU with 6GB memory.
 
+To run inference with int8, please ensure you have installed accelerate and bitandbytes. You can install them with the following command:
+
+```bash
+pip install accelerate
+pip install bitsandbytes
+```
+
+Then you can run inference with int8 as follows:
+
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1", device_map="auto", load_in_8bit=True)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1", device_map='auto', torch_dtype=torch.float16, load_in_8bit=True)
+
 # infer
-inputs = tokenizer("Hello", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
 ## CPU Inference
 
 ```python
+import torch
+import transformers
 from transformers import AutoTokenizer, AutoModelForCausalLM
+
+MIN_TRANSFORMERS_VERSION = '4.25.1'
+
+# check transformers version
+assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'
+
 # init
-tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1")
-model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Chat-INCITE-2.8B-v1", torch_dtype=torch.bfloat16)
+tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1")
+model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-Instruct-INCITE-2.8B-v1", torch_dtype=torch.bfloat16)
 # infer
-inputs = tokenizer("<human>: Hello!\n<bot>:", return_tensors='pt').to(model.device)
-outputs = model.generate(**inputs, max_new_tokens=10, do_sample=True, temperature=0.8)
-output_str = tokenizer.decode(outputs[0])
+prompt = "Q: The capital of France is?\nA:"
+inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
+input_length = inputs.input_ids.shape[1]
+outputs = model.generate(
+    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
+)
+token = outputs.sequences[0, input_length:]
+output_str = tokenizer.decode(token)
 print(output_str)
+"""
+Paris
+"""
 ```
 
+Please note that since `LayerNormKernelImpl` is not implemented in fp16 for CPU, we use `bfloat16` for CPU inference.
+
 
 # Uses
 
@@ -123,7 +185,7 @@ Please refer to [togethercomputer/RedPajama-Data-1T](https://huggingface.co/data
 - **Hardware:** 8 A100
 - **Optimizer:** Adam
 - **Gradient Accumulations**: 1
-- **Num of Tokens:** 1B Tokens
+- **Num of Tokens:** 131M tokens
 - **Learning rate:** 1e-5
 
 ## Community