Update tokenizer_config.json
#21
by
usoni1
- opened
Need to add bos_token, eos_token and unk_token explicitly. Otherwise, it causes maximum recursion depth error. See https://github.com/EleutherAI/lm-evaluation-harness/issues/442