The output of model is not correct and repeatly.
That's my code:
model_path = "codellama/CodeLlama-13b-Instruct-hf"
print(f"Starting to load the model {model_path} into memory")
tokenizer = CodeLlamaTokenizer.from_pretrained(model_path, use_fast=False)
model = LlamaForCausalLM.from_pretrained(
model_path, low_cpu_mem_usage=True, **from_pretrained_kwargs
).to("cuda")
qa = "please write a CUDA C kernel function example"
inputs = tokenizer.encode(qa, return_tensors="pt").to("cuda")
inputs_length = inputs.size()[1]
outputs = model.generate(inputs, temperature=0.7, do_sample=True, max_new_tokens=2048, top_k=1)
output = tokenizer.decode(outputs[0], skip_special_tokens=True, spaces_between_special_tokens=False, clean_up_tokenization_spaces=True)
print(output)
The output is as blow:
Result: please write a CUDA C kernel function example that does the same thing?
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA. I'm just trying to understand how to do this in C.
Comment: @RobertCrovella: I'm not sure what you mean by "kernel function". I'm not familiar with CUDA.
Could you guys help me to figure out what happens? Thank you in advance!
same problem. How do you solve it?@Shouyang
Try this prompt, note the [INST]:
[INST]
please write a CUDA C kernel function example
[/INST]
@tomredman
[Inst] works well with no problem!