croissantllm

/

CroissantLLMChat-v0.1

like 47

Follow

CroissantLLM 47

Salut,
Merci du message. C'est quoi la diff avec le tokenizer.json de CroissantLLMBase/tokenizer.json ?
C'est le même fichier ? On a pas changé le tokenizer entre les differentes versions.
Sinon on va te le faire rapidement,
Bien à toi,

Salut,
Honnêtement je ne sais pas… mise a part que le .model est un format binaire visiblement indispensable pour encoder et décoder. J'ai essayé toute la journée, et au moment de convertir le fichier au format FP16, le même message d'erreur est apparu.
(" Loading model file CroissantLLMChat-v01-hf/model-00001-of-00002.safetensors
Loading model file CroissantLLMChat-v01-hf/model-00001-of-00002.safetensors
Loading model file CroissantLLMChat-v01-hf/model-00002-of-00002.safetensors
params = Params(n_vocab=32002, n_embd=2048, n_layer=24, n_ctx=2048, n_ff=5504, n_head=16, n_head_kv=16, n_experts=None, n_experts_used=None, f_norm_eps=1e-05, rope_scaling_type=None, f_rope_freq_base=10000.0, f_rope_scale=None, n_orig_ctx=None, rope_finetuned=None, ftype=<GGMLFileType.MostlyF16: 1>, path_model=PosixPath('CroissantLLMChat-v01-hf'))
Found vocab files: {'tokenizer.model': PosixPath('CroissantLLMChat-v01-hf/tokenizer.model'), 'vocab.json': None, 'tokenizer.json': None}
Loading vocab file 'CroissantLLMChat-v01-hf/tokenizer.model', type 'spm'
Traceback (most recent call last):
File "/content/llama.cpp/convert.py", line 1474, in
main()
File "/content/llama.cpp/convert.py", line 1442, in main
vocab, special_vocab = vocab_factory.load_vocab(args.vocab_type, model_parent_path)
File "/content/llama.cpp/convert.py", line 1328, in load_vocab
vocab = SentencePieceVocab(
File "/content/llama.cpp/convert.py", line 394, in init
self.sentencepiece_tokenizer = SentencePieceProcessor(str(fname_tokenizer))
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/init.py", line 447, in Init
self.Load(model_file=model_file, model_proto=model_proto)
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/init.py", line 905, in Load
return self.LoadFromFile(model_file)
File "/usr/local/lib/python3.10/dist-packages/sentencepiece/init.py", line 310, in LoadFromFile
return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
RuntimeError: Internal: src/sentencepiece_processor.cc(1101) [model_proto->ParseFromArray(serialized.data(), serialized.size())] ")

J'ai fais quelques recherches, et il semblerait qu'il soit impossible de convertir le .json en .model avec sentencepiece sans ré-entrainer le modèle, or je n'ai pas la puissance nécessaire pour le faire moi même…
Merci en tout cas pour ta réponse rapide 👍

On va regarder rapidememt mais il y a un fichier ici : https://huggingface.co/croissantllm/CroissantLLMBase/blob/main/tokenizer.json c'est pas bon ?

On va regarder rapidememt mais il y a un fichier ici : https://huggingface.co/croissantllm/CroissantLLMBase/blob/main/tokenizer.json c'est pas bon ?

Merci beaucoup, si vous pouvez également le faire avec la version de base ce serait beaucoup plus simple pour moi d'éditer des GGUF et de commencer à les entrainements ! 🤗

Bonjour,

Serait il possible d'avoir une version .model du fichier tokenizer.json ?

Effectivement, s'il est possible d'essayer croissant LLM sur HF, il est en revanche impossible d'éditer un GGUF. Le fichier en .json n'est pas lu par sentencepiece ce qui est dommage...

En vous remerciant !

Bonjour @manu , je me permets de te tagguer, car j’ai cru comprendre que tu pourrais m’aider 😅

Désolé, j'avais pas vu...
https://huggingface.co/manu/CroissantLLMChat-v0.1-GGUF , et https://huggingface.co/manu/CroissantLLMBase-GGUF si tu veux des versions GGUF. (--vocab-type hfft flag dans llamacpp)
Je sais pas trop à quoi ca correspond les fichiers .model, ils sont ici au format standard HuggingFace donc tu peux les loader avec AutoTokenizer et les dumper comme tu veux...

Deprecated:

Je peux te passer ca aussi qui permet de convertir en GGUF:
https://colab.research.google.com/drive/17HSA0cawrJ4J99Mr62bnkLdP4u215vg7?usp=sharing
Headsup, dans le tokenizer y a 2 tokens qui sont mal traités par LlamaCPP, qui les lit comme des empty strings, donc ca peut poser des petits soucis maybe, que j'avais pas avec MLC AI ! 

Fix:

Fixed GGUF converter:
https://gist.githubusercontent.com/ManuelFay/a75cffecb77f6180f5dc2cd9c838f21c/raw/8cb8d57ebafa8a028aa1402321e1c3e5c90ce026/convert.py
https://colab.research.google.com/drive/1_jUDD-7PQWROv4MZSu5xiFOAAvGe2ywd?usp=sharing

Good luck,
Manu

[deprecated - see below]

Hey @Woziii ! Finally put in the time to figure out what was wrong with llamacpp model loading, it was 2 hours I probably should have spent sooner but was in so much rush these last few weeks ...
If it works out of the box for you great, if not, there's a few hacky things that can be done to patch things until I find a more stable solution !

In llama.cpp:
Replace line 3219:

        // GGML_ASSERT(codepoints_from_utf8(word).size() > 0);
        if (codepoints_from_utf8(word).empty()) {
                    std::stringstream ss;
                    for (unsigned char c : word) {  // Ensure char is treated as unsigned
                        ss << std::hex << static_cast<int>(c) << " ";  // Convert each byte to hex
                    }
                    LLAMA_LOG_WARN("%s: Word '%s' at position %d could not be converted to UTF-8 codepoints and will be replaced with unk+id\n", __func__, ss.str().c_str(), i);
                    if (i == 104) {
                        word = "<unk_104>";
                    } else {
                        word = "<unk_19078>";
                    }
                }

and line 3244

vocab.linefeed_id = 106; //llama_byte_to_token(vocab, '\n');

In short, our tokenizer is not exactly the same as Llama's, also in terms of construction rules and some edge cases were making tokenizer loading crash. Should be good now !
Cheers,
Manu

Heyyy @manu !
Don't worry, I understand completely, it's the same for me!

Actually, the error code I mentioned comes from the same GGUF converter that I had modified to avoid going through the HF hub. Yet, your solution is interesting! I had the same deduction but it was impossible for me to patch it...

I am going to try your patch today (si j'arrive à me libérer du temps au travail) or this weekend! I hope it works!

I'm staying tuned!

Thanks,

Lucas

Yeah for the conversion just the --vocab-type hfft should be enough with the convert script like is done in the notebook, no need to go through the hub !

Ok I find it !

But I'm gonna make a new convert script to work around the tokenizer issues that prevent "default" loading cause of the differences with llama !
It will be easier to have a GGUF version that is fully compatible than to have recompile llamacpp or make a custom architecture !

Great, thanks, I've identified the lines that need to be modified, and while I'm waiting for your script I'll also be trying things out on my own this weekend if I can find the time!

Hey !
I just patched the converter.py file directly, so the GGUF files run without any modifications to the CPP code, much cleaner this way:

Fixed GGUF converter:
https://gist.githubusercontent.com/ManuelFay/a75cffecb77f6180f5dc2cd9c838f21c/raw/8cb8d57ebafa8a028aa1402321e1c3e5c90ce026/convert.py

Colab notebook to quantize etc:
https://colab.research.google.com/drive/1_jUDD-7PQWROv4MZSu5xiFOAAvGe2ywd?usp=sharing

The llamacpp code works out of the box like this ! Essentially, I changed the \n token to b"<0x0A>" and I fixed the other two tokens that were problematic at conversion time !
I'll upload the GGUF files + fp16.bin files in this organization !
Thanks again for the interest !

Uploaded !