TheBloke

/

Yarn-Llama-2-13B-128K-GPTQ

like 16

Transformers, optimum, flash_attn all installed from source

(torch) arda@toprak:/data/arda/projects/test/Yarn-Llama-2-13B-128K-GPTQ$ python main.py 
Traceback (most recent call last):
  File "/data/arda/projects/test/Yarn-Llama-2-13B-128K-GPTQ/main.py", line 7, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_name_or_path,
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/arda/envs/torch/lib/python3.11/site-packages/transformers/models/auto/auto_factory.py", line 519, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/arda/envs/torch/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1037, in from_pretrained
    return config_class.from_dict(config_dict, **unused_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/arda/envs/torch/lib/python3.11/site-packages/transformers/configuration_utils.py", line 747, in from_dict
    config = cls(**config_dict)
             ^^^^^^^^^^^^^^^^^^
  File "/data/arda/envs/torch/lib/python3.11/site-packages/transformers/models/llama/configuration_llama.py", line 149, in __init__
    self._rope_scaling_validation()
  File "/data/arda/envs/torch/lib/python3.11/site-packages/transformers/models/llama/configuration_llama.py", line 167, in _rope_scaling_validation
    raise ValueError(
ValueError: `rope_scaling` must be a dictionary with with two fields, `type` and `factor`, got {'factor': 32.0, 'original_max_position_embeddings': 4096, 'type': 'yarn', 'finetuned': True}
(torch) arda@toprak:/data/arda/projects/test/Yarn-Llama-2-13B-128K-GPTQ$