maddes8cht
commited on
Commit
•
81d35d0
1
Parent(s):
fe2124c
"Update README.md"
Browse files
README.md
CHANGED
@@ -9,21 +9,7 @@ I'm constantly enhancing these model descriptions to provide you with the most r
|
|
9 |
- Model creator: [adept](https://huggingface.co/adept)
|
10 |
- Original model: [persimmon-8b-base](https://huggingface.co/adept/persimmon-8b-base)
|
11 |
|
12 |
-
|
13 |
-
|
14 |
-
As noted on the [Llama.cpp GitHub repository](https://github.com/ggerganov/llama.cpp#hot-topics), all new Llama.cpp releases after October 18, 2023, will require a re-quantization due to the new BPE tokenizer.
|
15 |
-
|
16 |
-
**Good news!** I am glad that my re-quantization process for Falcon Models is nearly complete. Download the latest quantized models to ensure compatibility with recent llama.cpp software.
|
17 |
-
|
18 |
-
**Key Points:**
|
19 |
-
|
20 |
-
- **Stay Informed:** Keep an eye on software application release schedules using llama.cpp libraries.
|
21 |
-
- **Monitor Upload Times:** Re-quantization is *almost* done. Watch for updates on my Hugging Face Model pages.
|
22 |
-
|
23 |
-
**Important Compatibility Note:** Old software will work with old Falcon models, but expect updated software to exclusively support the new models.
|
24 |
-
|
25 |
-
This change primarily affects **Falcon** and **Starcoder** models, with other models remaining unaffected.
|
26 |
-
|
27 |
|
28 |
|
29 |
|
@@ -35,19 +21,21 @@ The core project making use of the ggml library is the [llama.cpp](https://githu
|
|
35 |
|
36 |
# Quantization variants
|
37 |
|
38 |
-
There is a bunch of quantized files available.
|
39 |
|
40 |
# Legacy quants
|
41 |
|
42 |
Q4_0, Q4_1, Q5_0, Q5_1 and Q8 are `legacy` quantization types.
|
43 |
Nevertheless, they are fully supported, as there are several circumstances that cause certain model not to be compatible with the modern K-quants.
|
44 |
-
|
|
|
|
|
45 |
|
46 |
# K-quants
|
47 |
|
48 |
-
K-quants are
|
49 |
So, if possible, use K-quants.
|
50 |
-
With a Q6_K you
|
51 |
|
52 |
|
53 |
|
|
|
9 |
- Model creator: [adept](https://huggingface.co/adept)
|
10 |
- Original model: [persimmon-8b-base](https://huggingface.co/adept/persimmon-8b-base)
|
11 |
|
12 |
+
Persimmon is a Large language Model from Adept AI. It is trained from Scratch with a context legth of 16k, which is 4 times the context size of LLaMA2 and 8 times that of GPT-3
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
13 |
|
14 |
|
15 |
|
|
|
21 |
|
22 |
# Quantization variants
|
23 |
|
24 |
+
There is a bunch of quantized files available to cater to your specific needs. Here's how to choose the best option for you:
|
25 |
|
26 |
# Legacy quants
|
27 |
|
28 |
Q4_0, Q4_1, Q5_0, Q5_1 and Q8 are `legacy` quantization types.
|
29 |
Nevertheless, they are fully supported, as there are several circumstances that cause certain model not to be compatible with the modern K-quants.
|
30 |
+
## Note:
|
31 |
+
Now there's a new option to use K-quants even for previously 'incompatible' models, although this involves some fallback solution that makes them not *real* K-quants. More details can be found in affected model descriptions.
|
32 |
+
(This mainly refers to Falcon 7b and Starcoder models)
|
33 |
|
34 |
# K-quants
|
35 |
|
36 |
+
K-quants are designed with the idea that different levels of quantization in specific parts of the model can optimize performance, file size, and memory load.
|
37 |
So, if possible, use K-quants.
|
38 |
+
With a Q6_K, you'll likely find it challenging to discern a quality difference from the original model - ask your model two times the same question and you may encounter bigger quality differences.
|
39 |
|
40 |
|
41 |
|