TypeError: __init__() got an unexpected keyword argument 'bias'
I managed to download the model, but then when I load it with
model = AutoModelForCausalLM.from_pretrained(checkpoint, cache_dir=cache_dir, torch_dtype=torch.bfloat16, device_map='auto', trust_remote_code=True)
I get the following error:
[...]
self.norm_1 = nn.LayerNorm(hidden_size, bias=False)
TypeError: init() got an unexpected keyword argument 'bias'
I'm using torch version '2.0.0+rocm5.4.2' and transformers version '4.39.2'. Any idea on how to solve it?
I had the same problem - i got past this error.. try installing torch 2.2.2 that worked for me. Also i am using virtualenv, try if that works for you. i haven't got it fully working either.. atleast its not this error
Looking at the latest PyTorch docs for nn.LayerNorm here, it is clear that nn.LayerNorm accepts the parameter bias. Make sure you are using the latest version of torch and try again. Let us know how that goes...