Upload README.md

README.md

@@ -20,8 +20,6 @@ widget:
 
 Original model: https://huggingface.co/google/gemma-2-2b-jpn-it
 
-Run them in [LM Studio](https://lmstudio.ai/)
-
 ## Prompt format
 
 ```
@@ -30,49 +28,68 @@ Run them in [LM Studio](https://lmstudio.ai/)
 
 ## Download a file (not the whole branch) from below:
 
+ELIZA-Tasks-100 is pretty standard benchmark for Japanese LLMs. 
+The perfect score is 5.00. As a reference, bartowski's gemma-2-27b-it.Q6_K.gguf scores 4.04.
+
 | Filename | Quant type | File Size | Split | ELIZA-Tasks-100 | Description |
 | -------- | ---------- | --------- | ----- | --------------- | ----------- |
 | [gemma-2-2b-jpn-it.f16.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.f16.gguf) | f16 | 5.24GB | false | Full F16 weights. |
-| [gemma-2-2b-jpn-it.Q8_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q8_0.gguf) | Q8_0 | 2.78GB | false | Extremely high quality, *recommended*. |
-| [gemma-2-2b-jpn-it-imatrix.Q4_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0.gguf) | Q4_0 | 1.63GB | false | Good quality, *recommended for edge device <8GB RAM*. |
-| [gemma-2-2b-jpn-it-imatrix.Q4_0_8_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_8_8.gguf) | Q4_0_8_8 | 1.63GB | false | Good quality, *recommended for edge device <8GB RAM*. |
-| [gemma-2-2b-jpn-it-imatrix.Q4_0_4_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_4_8.gguf) | Q4_0_4_8 | 1.63GB | false | Good quality, *recommended for edge device <8GB RAM*. |
-| [gemma-2-2b-jpn-it-imatrix.Q4_0_4_4.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_4_4.gguf) | Q4_0_4_4 | 1.63GB | false | Good quality, *recommended for edge device <8GB RAM*. |
-| [gemma-2-2b-jpn-it.Q4_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0.gguf) | Q4_0 | 1.63GB | false | Poor quality, *not recommended*. |
-| [gemma-2-2b-jpn-it.Q4_0_8_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_8_8.gguf) | Q4_0_8_8 | 1.63GB | false | Poor quality, *not recommended*. |
-| [gemma-2-2b-jpn-it.Q4_0_4_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_4_8.gguf) | Q4_0_4_8 | 1.63GB | false | Poor quality, *not recommended*. |
-| [gemma-2-2b-jpn-it.Q4_0_4_4.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_4_4.gguf) | Q4_0_4_4 | 1.63GB | false | Poor quality, *not recommended*. |
+| [gemma-2-2b-jpn-it.Q8_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q8_0.gguf) | Q8_0 | 2.78GB | false | 3.06 | Extremely high quality, *recommended*. |
+| [gemma-2-2b-jpn-it-imatrix.Q4_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0.gguf) | Q4_0 | 1.63GB | false | 2.89 | Good quality, *recommended for edge device <8GB RAM*. |
+| [gemma-2-2b-jpn-it-imatrix.Q4_0_8_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_8_8.gguf) | Q4_0_8_8 | 1.63GB | false | 2.78 | Good quality, *recommended for edge device <8GB RAM*. |
+| [gemma-2-2b-jpn-it-imatrix.Q4_0_4_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_4_8.gguf) | Q4_0_4_8 | 1.63GB | false | TBD | Good quality, *recommended for edge device <8GB RAM*. |
+| [gemma-2-2b-jpn-it-imatrix.Q4_0_4_4.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it-imatrix.Q4_0_4_4.gguf) | Q4_0_4_4 | 1.63GB | false | TBD | Good quality, *recommended for edge device <8GB RAM*. |
+| [gemma-2-2b-jpn-it.Q4_0.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0.gguf) | Q4_0 | 1.63GB | false | 2.77 | Good quality but imatrix version a bit better. |
+| [gemma-2-2b-jpn-it.Q4_0_8_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_8_8.gguf) | Q4_0_8_8 | 1.63GB | false | TBD | Poor quality, *not recommended*. |
+| [gemma-2-2b-jpn-it.Q4_0_4_8.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_4_8.gguf) | Q4_0_4_8 | 1.63GB | false | TBD | Poor quality, *not recommended*. |
+| [gemma-2-2b-jpn-it.Q4_0_4_4.gguf](https://huggingface.co/ymcki/gemma-2-2b-jpn-it-GGUF/blob/main/gemma-2-2b-jpn-it.Q4_0_4_4.gguf) | Q4_0_4_4 | 1.63GB | false | TBD | Poor quality, *not recommended*. |
 
 ## How to check i8mm and sve support for ARM devices
 
-ARM i8mm support is necessary to take advantage of Q4_0_4_8 gguf. All ARM architecure >= ARMv8.6-A supports i8mm.
+ARM i8mm support is necessary to take advantage of Q4_0_4_8 gguf. All ARM architecture >= ARMv8.6-A supports i8mm.
 
 ARM sve support is necessary to take advantage of Q4_0_8_8 gguf. sve is an optional feature that starts from ARMv8.2-A but majority of ARM chips doesn't implement it.
 
 For ARM devices without both, it is recommended to use Q4_0_4_4.
 
+With these support, the inference speed should be faster in the order of Q4_0_8_8 > Q4_0_4_8 > Q4_0_4_4 > Q4_0 without much effect on the quality of response.
+
+
+This is a [list](https://gpages.juszkiewicz.com.pl/arm-socs-table/arm-socs.html) of ARM devices that support different ARM instructions. Apparently, it is only a partial list. It is better you check for i8mm and sve support by yourself.
+
 For Apple devices, 
 
 ```
 sysctl hw
 ```
 
-For ARM devices (ie most Android devices),
+For other ARM devices (ie most Android devices),
 ```
 cat /proc/cpuinfo
 ```
 
 There are also android apps that can display /proc/cpuinfo.
 
+On the other hand, inference speed for the imatrix ggufs on my Nvidia 3090 is 137t/s for Q4_0, 2.8t/s for Q4_0_8_8. That means for Nvidia, you better off using Q4_0. 
+
 ## Which Q4_0 model to use for ARM devices
 | Brand | Series | Model | i8mm | sve | Quant Type |
 | ----- | ------ | ----- | ---- | --- | -----------|
-| Qualcomm ｜Snapdragon | >= 7 Gen 1 | Yes | Yes | Q4_0_8_8 |
-| Qualcomm ｜Snapdragon | others | No | No | Q4_0_4_4 |
-| Apple | M | M1 | No | No | Q4_0_4_4 |
-| Apple | M | M2/M3/M4 | Yes | No | Q4_0_4_8 |
 | Apple | A | A4 to A14 | No | No | Q4_0_4_4 |
 | Apple | A | A15 to A18 | Yes | No | Q4_0_4_8 |
+| Apple | M | M1 | No | No | Q4_0_4_4 |
+| Apple | M | M2/M3/M4 | Yes | No | Q4_0_4_8 |
+| Google | Tensor | G1,G2 | No | No | Q4_0_4_4 |
+| Google | Tensor | G3,G4 | Yes | Yes | Q4_0_8_8 |
+| Samsung | Exynos | 2200,2400 | Yes | Yes | Q4_0_8_8 |
+| Mediatek | Dimensity | 9000 | Yes | Yes | Q4_0_8_8 |
+| Mediatek | Dimensity | 9300 | Yes | No | Q4_0_4_8 |
+| Qualcomm ｜Snapdragon | 8 Gen 1 | Yes | Yes | Q4_0_8_8 |
+| Qualcomm ｜Snapdragon | 8 Gen 2,8 Gen 3,X Elite | Yes | No | Q4_0_4_8 |
+
+## imatrix quantization
+
+According to this [blog](https://sc-bakushu.hatenablog.com/entry/2024/04/20/050213), adding imatrix to low bit quant can significantly improve performance. The best dataset for Japanese is [MTFMC/imatrix-dataset-for-japanese-llm](https://huggingface.co/datasets/TFMC/imatrix-dataset-for-japanese-llm). Therefore, I also created the imatrix versions of different Q4_0 quants. Indeed, they significantly outperforms the non-imatrix counterparts.
 
 ## Convert safetensors to f16 gguf
 
@@ -88,7 +105,7 @@ Make sure you have llama.cpp compiled:
 ./llama-quantize gemma-2-2b-jpn-it.f16.gguf gemma-2-2b-jpn-it.Q8_0.gguf q8_0
 ```
 
-## Convert f16 gguf to other gguf with imatrix
+## Convert f16 gguf to other ggufs with imatrix
 
 First, prepare imatrix from f16 gguf and c4_en_ja_imatrix.txt