TheFloat16

/

Llama3-70b-Instruct-TRTLLM

like 2

When I try to load one of these models, I get:

│ /home/user/miniconda3/envs/tensorrt/lib/python3.10/site-packages/tensorrt_llm/runtime/model_runner │
│ .py:85 in _builder_to_model_config                                                                    │
│                                                                                                       │
│    84 def _builder_to_model_config(config: dict) -> Tuple[ModelConfig, dict]:                         │
│ ❱  85     builder_config = config['builder_config']                                                   │
│    86     model_name = builder_config['name']                                                         │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'builder_config'

This happens because the structure of config.json has changed from 0.9 to 0.10. It would be nice if you could update the config.json files in this repository.