dreamgen/opus-llama-3-tokens-alt

What is this?

This is alternative configuration for some special tokens for Opus Llama 3 models. This is needed for some backends because of the issues described here:

• https://huggingface.co/dreamgen/opus-v1.2-llama-3-8b/discussions/3
• https://github.com/ggerganov/llama.cpp/issues/6770

What are the changes?

First, DreamGen Opus models use a variant of ChatML, so we rename <|start_header_id|> to <|im_start|> and <|eot_id|> to <|im_end|>. This is already done in the DreamGen Opus Llama 3 fp16 repos.

Then, in order to address the issues some backends are having with Llama 3's special tokens:

• We set "special": false for both <|im_start|> and <|im_end|> in various places. This will allow them to be rendered by some frontends. Originally discovered here.
• We set the EOS token to <|im_end|> in various places.

I consider using <|im_end|> as a stop token as suboptimal, as in multi-character scenarios I like to let he model generate multiple character messages at once, and this prevents that. But for now, until we can get custom stop strings with special tokens working, this is the best we have.