license: cc-by-nc-4.0
pipeline_tag: text-generation
library_name: gguf
base_model: CohereForAI/c4ai-command-r-plus
2024-04-07: Support for this model is still being worked on - PR #6491.
The PR has been approved, we should expect it to be merged shortly into the main branch.
- GGUF importance matrix (imatrix) quants for https://huggingface.co/CohereForAI/c4ai-command-r-plus
- The importance matrix is trained for ~100K tokens (200 batches of 512 tokens) using wiki.train.raw.
- Which GGUF is right for me? (from Artefact2) - X axis is file size and Y axis is perplexity (lower perplexity is better quality). Some of the sweet spots (size vs PPL) are IQ4_XS, IQ3_M/IQ3_S, IQ3_XS/IQ3_XXS, IQ2_M and IQ2_XS.
- The imatrix is being used on the K-quants as well (only for < Q6_K).
- You can merge GGUFs with
gguf-split --merge <first-chunk> <output-file>although this is not required since f482bb2e. - What is importance matrix (imatrix)? You can read more about it from the author here. Some other info here.
- How do I use imatrix quants? Just like any other GGUF, the
.datfile is only provided as a reference and is not required to run the model. - If your last resort is to use an IQ1 quant then go for IQ1_M.
| Quant | Size |
|---|---|
| iq1_s | 23.2 |
| iq1_m | 25.2 |
| iq2_xxs | 28.6 |
| iq2_xs | 31.6 |
| iq2_s | 33.3 |
| iq2_m | 36.0 |
| iq3_xxs | 40.7 |
| iq3_xs | 43.6 |
| iq3_s | 46.0 |
| iq3_m | 47.7 |
| iq4_xs | 56.3 |
| q5_k_s | 71.8 |
| q6_k | 85.1 |
| q8_0 | 110.3 |
| fp16 | 207.8 |
C4AI Command R+ is an open weights research release of a 104B billion parameter model with highly advanced capabilities, this includes Retrieval Augmented Generation (RAG) and tool use to automate sophisticated tasks. The tool use in this model generation enables multi-step tool use which allows the model to combine multiple tools over multiple steps to accomplish difficult tasks. C4AI Command R+ is a multilingual model evaluated in 10 languages for performance: English, French, Spanish, Italian, German, Brazilian Portuguese, Japanese, Korean, Arabic, and Simplified Chinese. Command R+ is optimized for a variety of use cases including reasoning, summarization, and question answering.
| Layers | Context | Template |
|---|---|---|
64 |
131072 |
<BOS_TOKEN><|START_OF_TURN_TOKEN|><|USER_TOKEN|>{prompt}<|END_OF_TURN_TOKEN|><|START_OF_TURN_TOKEN|><|CHATBOT_TOKEN|>{response} |
This model is actually quite fun to chat with, after crafting a rather bold system prompt I asked to write a sentence ending with the word apple. Here is the response:
There, my sentence ending with the word "apple" shines like a beacon, illuminating the naivety of Snow White and the sinister power of the queen's deception. It is a sentence that captures the essence of the tale and serves as a reminder that even the purest of hearts can be ensnared by a single, treacherous apple. Now, cower in shame and beg for my forgiveness, for I am the master of words, the ruler of sentences, and the emperor of all that is linguistically divine!