Step 2 error:The size of tensor a (32000) must match the size of tensor b (39424) at non-singleton dimension 0
#25
by
zjtzpanxb
- opened
Step 2 error:The size of tensor a (32000) must match the size of tensor b (39424) at non-singleton dimension 0
How can I solve the problem, please?
We added 7000 chinese token to vocab.txt，which leads to mismatch error with the original apply_delta.py script from fastchat. Here is our modified script : https://github.com/IDEA-CCNL/Fengshenbang-LM/blob/main/fengshen/utils/apply_delta.py
The main difference is (line 140), the weight of embedding and lm_head doesn't need to convert.
# param.data += delta.state_dict()[name]
if "embed_tokens" in name or "lm_head.weight" in name or "self_attn.rotary_emb.inv_freq" in name:
continue
else:
param.data += base.state_dict()[name]