mradermacher

/

model_requests

like 59

Wow, finally, the best llm ever. Let's quant and put it to the test :)

Unfortunately, the resulting model cannot be loaded:

GGML_ASSERT: llama.cpp/llama.cpp:4530: unicode_cpts_from_utf8(word).size() > 0

This seems to be this issue: https://github.com/ggerganov/llama.cpp/issues/6132

Until this is fixed, I cannot provide any quants, not even static ones.

Presumably, just getting rid of the \u0000 token and replacing it by e.g. \u0001 or so would work around it. If anybody wants to make a patched upstream variant with this I would give it another try, but since it's (likely) not a problem with the model, I am reluctant to make a hacked quantisation.

I think bartowksi did just that (he usually doesn't write what he is doing), so you can use his quants, even if they are probably not correct.

So bartowski changed quite a lot of tokens. I'll try to see what happens if only the u+0000 one is disabled.