Update README.md
Browse files
README.md
CHANGED
|
@@ -14,7 +14,7 @@ tags:
|
|
| 14 |
|
| 15 |
## Overview
|
| 16 |
|
| 17 |
-
The **Fhi-3.5-mini-instruct** is a fine-tuned version of the [unsloth/Phi-3.5-mini-instruct](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) model, optimized for function-calling
|
| 18 |
|
| 19 |
### Usage
|
| 20 |
|
|
@@ -64,5 +64,25 @@ The result will look like this:
|
|
| 64 |
## Testing and Benchmarking
|
| 65 |
This model is still undergoing testing and evaluation. Use it at your own risk until further validation is complete. Performance on benchmarks like MMLU and MMLU-Pro will be updated soon.
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## Credits
|
| 68 |
Will be updated soon
|
|
|
|
| 14 |
|
| 15 |
## Overview
|
| 16 |
|
| 17 |
+
The **Fhi-3.5-mini-instruct** is a fine-tuned version of the [unsloth/Phi-3.5-mini-instruct](https://huggingface.co/unsloth/Phi-3.5-mini-instruct) model, optimized for function-calling.
|
| 18 |
|
| 19 |
### Usage
|
| 20 |
|
|
|
|
| 64 |
## Testing and Benchmarking
|
| 65 |
This model is still undergoing testing and evaluation. Use it at your own risk until further validation is complete. Performance on benchmarks like MMLU and MMLU-Pro will be updated soon.
|
| 66 |
|
| 67 |
+
| Benchmark | Fhi-3.5 Mini-Ins | Phi-3.5 Mini-Ins | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
|
| 68 |
+
|----------------------------|------------------|------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
|
| 69 |
+
| Multilingual MMLU | ____ | 55.4 | 47.4 | 58.9 | 56.2 | 63.8 | 77.2 | 72.9 |
|
| 70 |
+
<!-- | Multilingual MMLU-Pro | 30.9 | 30.21 | 15.0 | 34.0 | 21.4 | 43.0 | 57.9 | 53.2 |
|
| 71 |
+
| **Average** | **55.2** | **52.3** | **47.9** | **55.3** | **47.5** | **59.6** | **64.3** | **76.6** |
|
| 72 |
+
-->
|
| 73 |
+
<!-- The table below shows Multilingual MMLU scores in some of the supported languages.
|
| 74 |
+
|
| 75 |
+
| Benchmark | Phi-3.5 Mini-Ins | Phi-3.0-Mini-128k-Instruct (June2024) | Mistral-7B-Instruct-v0.3 | Mistral-Nemo-12B-Ins-2407 | Llama-3.1-8B-Ins | Gemma-2-9B-Ins | Gemini 1.5 Flash | GPT-4o-mini-2024-07-18 (Chat) |
|
| 76 |
+
|-----------|------------------|-----------------------|--------------------------|---------------------------|------------------|----------------|------------------|-------------------------------|
|
| 77 |
+
| Arabic | 44.2 | 35.4 | 33.7 | 45.3 | 49.1 | 56.3 | 73.6 | 67.1 |
|
| 78 |
+
| Chinese | 52.6 | 46.9 | 45.9 | 58.2 | 54.4 | 62.7 | 66.7 | 70.8 |
|
| 79 |
+
| Dutch | 57.7 | 48.0 | 51.3 | 60.1 | 55.9 | 66.7 | 80.6 | 74.2 |
|
| 80 |
+
| French | 61.1 | 61.7 | 53.0 | 63.8 | 62.8 | 67.0 | 82.9 | 75.6 |
|
| 81 |
+
| German | 62.4 | 61.3 | 50.1 | 64.5 | 59.9 | 65.7 | 79.5 | 74.3 |
|
| 82 |
+
| Italian | 62.8 | 63.1 | 52.5 | 64.1 | 55.9 | 65.7 | 82.6 | 75.9 |
|
| 83 |
+
| Russian | 50.4 | 45.3 | 48.9 | 59.0 | 57.4 | 63.2 | 78.7 | 72.6 |
|
| 84 |
+
| Spanish | 62.6 | 61.3 | 53.9 | 64.3 | 62.6 | 66.0 | 80.0 | 75.5 |
|
| 85 |
+
| Ukrainian | 45.2 | 36.7 | 46.9 | 56.6 | 52.9 | 62.0 | 77.4 | 72.6 |
|
| 86 |
+
-->
|
| 87 |
## Credits
|
| 88 |
Will be updated soon
|