neuralmagic
/

Meta-Llama-3-8B-Instruct-quantized.w4a16

Text Generation

text-generation-inference

Inference Endpoints

4-bit precision

Model card Files Files and versions Community

abhinavnmagic commited on Jul 16

Commit

1769a68

•

1 Parent(s): 0ce6732

Update README.md

Files changed (1) hide show

README.md +1 -1

README.md CHANGED Viewed

@@ -51,7 +51,7 @@ messages = [
     {"role": "user", "content": "Who are you?"},
 ]
-prompts = tokenizer.apply_chat_template(messages, tokenize=False)
 llm = LLM(model=model_id, tensor_parallel_size=1)

     {"role": "user", "content": "Who are you?"},
 ]
+prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
 llm = LLM(model=model_id, tensor_parallel_size=1)