The model outputs `!!!!!!!!!!!!!!!!!!!!` only
#1
by
seungduk
- opened
Hi there,
Thanks for sharing a good model. I wanted to test it out but I failed to get the expected output from the model. Can you point me out what I did wrong?
Code:
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "nayohan/llama3-instrucTrans-enko-8b"
model = AutoModelForCausalLM.from_pretrained(
model_id,
torch_dtype=torch.bfloat16,
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)
system_prompt = "๋น์ ์ ๋ฒ์ญ๊ธฐ ์
๋๋ค. ์์ด๋ฅผ ํ๊ตญ์ด๋ก ๋ฒ์ญํ์ธ์."
sentence = "The aerospace industry is a flower in the field of technology and science."
conversation = [{'role': 'system', 'content': system_prompt},
{'role': 'user', 'content': sentence}]
encoded_inputs = tokenizer.apply_chat_template(
conversation,
tokenize=True,
add_generation_prompt=True,
return_tensors='pt'
)
input_ids = encoded_inputs.to('cuda')
attention_mask = torch.ones_like(input_ids)
with torch.no_grad():
outputs = model.generate(input_ids=input_ids, attention_mask=attention_mask
, max_new_tokens=46) # Finetuned with length 4096
print(tokenizer.decode(outputs[0]))
Output:
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
๋น์ ์ ๋ฒ์ญ๊ธฐ ์
๋๋ค. ์์ด๋ฅผ ํ๊ตญ์ด๋ก ๋ฒ์ญํ์ธ์.<|eot_id|><|start_header_id|>user<|end_header_id|>
The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Thanks,
Seungduk
I can not re-generate your problems.
In my case, it works beautifully.
Loading checkpoint shards: 100%|โโโโโโโโโโ| 4/4 [00:01<00:00, 2.39it/s]
WARNING:root:Some parameters are on the meta device device because they were offloaded to the cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
๋น์ ์ ๋ฒ์ญ๊ธฐ ์
๋๋ค. ์์ด๋ฅผ ํ๊ตญ์ด๋ก ๋ฒ์ญํ์ธ์.<|eot_id|><|start_header_id|>user<|end_header_id|>
The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
ํญ๊ณต์ฐ์ฃผ ์ฐ์
์ ๊ธฐ์ ๊ณผ ๊ณผํ ๋ถ์ผ์ ๊ฝ์
๋๋ค.<|eot_id|>
I also got the same result as above. even though I used your fixed code.
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
๋น์ ์ ๋ฒ์ญ๊ธฐ ์
๋๋ค. ์์ด๋ฅผ ํ๊ตญ์ด๋ก ๋ฒ์ญํ์ธ์.<|eot_id|><|start_header_id|>user<|end_header_id|>
The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>
ํญ๊ณต์ฐ์ฃผ ์ฐ์
์ ๊ธฐ์ ๊ณผ ๊ณผํ ๋ถ์ผ์ ๊ฝ์
๋๋ค.<|eot_id|>
I'll share my environment settings, would you like to try this?
--find-links https://download.pytorch.org/whl/torch_stable.html
torch==2.2.1+cu121
transformers==4.40.1
datasets==2.18.0
accelerate==0.26.1
deepspeed==0.14.0
peft==0.10.0
trl==0.8.6
pyarrow==15.0.0
# flash-attn==2.5.6
bitsandbytes==0.43.1
My guess is that you installed the package using pip
. Problems caused by pip
are ambiguous and hard to solve. If you can, try installing using the conda
package manager and the conda-forge
channel.