Lack of input_layernorm layers?
First, thank you for your work on fine-tuning new models and also quantizing them for use with exllama2, great stuff!
I downloaded this model and I am trying to run this with exllama2 on Windows.
Running it with test_inference.py works fine, but I receive an error when trying to run chat
python examples\chat.py -m models\yi6b-spicyboros -mode raw
-- Model: models\yi6b-spicyboros
-- Options: ['rope_scale 1.0', 'rope_alpha 1.0']
Traceback (most recent call last):
File "C:\Users\adamo\maked\exllama\exllamav2\examples\chat.py", line 81, in <module>
model, tokenizer = model_init.init(args, allow_auto_split = True)
File "C:\Users\adamo\AppData\Local\Programs\Python\Python310\lib\site-packages\exllamav2\model_init.py", line 64, in init
config.prepare()
File "C:\Users\adamo\AppData\Local\Programs\Python\Python310\lib\site-packages\exllamav2\config.py", line 133, in prepare
raise ValueError(f" ## Could not find {prefix}.* in model")
ValueError: ## Could not find model.layers.0.input_layernorm.* in model
I printed the list of layers for this model (variable self.tensor_file_map from config.py) and indeed I can't find any input_layernorm layers there.
I checked for the same issue with EXL2 quantized TinyLlama from turboderp and this issue does not exist there.
Am I missing something obvious?
I quantized using the latest exl2 git repo and am able to load it using ooba text gen and generate responses (though it still has issues stopping when generation is complete):
2023-11-06 22:23:14 INFO:Loading Yi-6B-Spicyboros-3.1-4.0bpw-h6-exl2...
2023-11-06 22:23:16 INFO:Loaded the model in 1.38 seconds.
Output generated in 15.38 seconds (99.69 tokens/s, 1533 tokens, context 95, seed 482399913)
I'll test with the command line interface when I get a chance later today.
I managed to solve the issue. I was running chat.py from up-to-date repo, but I was using pre-compiled 0.0.7 wheel and not self-complied version. The pre-compiled version didn't have commit 6d24e1ad40d89f64b1bd3ae36e639c74c9f730b2 that adds Yi compatibility.
I compiled the package with python setup.py install --user and now it chat.py works.
closing the issue