The model outputs `!!!!!!!!!!!!!!!!!!!!` only

#1
by seungduk - opened

Hi there,

Thanks for sharing a good model. I wanted to test it out but I failed to get the expected output from the model. Can you point me out what I did wrong?

Code:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "nayohan/llama3-instrucTrans-enko-8b"

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

system_prompt = "๋‹น์‹ ์€ ๋ฒˆ์—ญ๊ธฐ ์ž…๋‹ˆ๋‹ค. ์˜์–ด๋ฅผ ํ•œ๊ตญ์–ด๋กœ ๋ฒˆ์—ญํ•˜์„ธ์š”."
sentence = "The aerospace industry is a flower in the field of technology and science."
conversation = [{'role': 'system', 'content': system_prompt},
                {'role': 'user', 'content': sentence}]

encoded_inputs = tokenizer.apply_chat_template(
  conversation,
  tokenize=True,
  add_generation_prompt=True,
  return_tensors='pt'
)

input_ids = encoded_inputs.to('cuda')
attention_mask = torch.ones_like(input_ids)

with torch.no_grad():
    outputs = model.generate(input_ids=input_ids, attention_mask=attention_mask
                             , max_new_tokens=46) # Finetuned with length 4096
    print(tokenizer.decode(outputs[0]))

Output:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

๋‹น์‹ ์€ ๋ฒˆ์—ญ๊ธฐ ์ž…๋‹ˆ๋‹ค. ์˜์–ด๋ฅผ ํ•œ๊ตญ์–ด๋กœ ๋ฒˆ์—ญํ•˜์„ธ์š”.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Thanks,
Seungduk

I can not re-generate your problems.
In my case, it works beautifully.

Loading checkpoint shards: 100%|โ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆโ–ˆ| 4/4 [00:01<00:00,  2.39it/s]
WARNING:root:Some parameters are on the meta device device because they were offloaded to the cpu.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Setting `pad_token_id` to `eos_token_id`:128001 for open-end generation.
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

๋‹น์‹ ์€ ๋ฒˆ์—ญ๊ธฐ ์ž…๋‹ˆ๋‹ค. ์˜์–ด๋ฅผ ํ•œ๊ตญ์–ด๋กœ ๋ฒˆ์—ญํ•˜์„ธ์š”.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

ํ•ญ๊ณต์šฐ์ฃผ ์‚ฐ์—…์€ ๊ธฐ์ˆ ๊ณผ ๊ณผํ•™ ๋ถ„์•ผ์˜ ๊ฝƒ์ž…๋‹ˆ๋‹ค.<|eot_id|>
Owner
โ€ข
edited May 30

I also got the same result as above. even though I used your fixed code.

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

๋‹น์‹ ์€ ๋ฒˆ์—ญ๊ธฐ ์ž…๋‹ˆ๋‹ค. ์˜์–ด๋ฅผ ํ•œ๊ตญ์–ด๋กœ ๋ฒˆ์—ญํ•˜์„ธ์š”.<|eot_id|><|start_header_id|>user<|end_header_id|>

The aerospace industry is a flower in the field of technology and science.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

ํ•ญ๊ณต์šฐ์ฃผ ์‚ฐ์—…์€ ๊ธฐ์ˆ ๊ณผ ๊ณผํ•™ ๋ถ„์•ผ์˜ ๊ฝƒ์ž…๋‹ˆ๋‹ค.<|eot_id|>
Owner

I'll share my environment settings, would you like to try this?

--find-links https://download.pytorch.org/whl/torch_stable.html
torch==2.2.1+cu121
transformers==4.40.1
datasets==2.18.0
accelerate==0.26.1
deepspeed==0.14.0
peft==0.10.0
trl==0.8.6
pyarrow==15.0.0
# flash-attn==2.5.6
bitsandbytes==0.43.1

My guess is that you installed the package using pip. Problems caused by pip are ambiguous and hard to solve. If you can, try installing using the conda package manager and the conda-forge channel.

Sign up or log in to comment