Quantization made by Richard Erkhov. [Github](https://github.com/RichardErkhov) [Discord](https://discord.gg/pvy7H8DZMG) [Request more models](https://github.com/RichardErkhov/quant_request) Moza-7B-v1.0 - bnb 8bits - Model creator: https://huggingface.co/kidyu/ - Original model: https://huggingface.co/kidyu/Moza-7B-v1.0/ Original model description: --- license: apache-2.0 library_name: transformers tags: - mergekit - merge base_model: - mistralai/Mistral-7B-v0.1 - cognitivecomputations/dolphin-2.2.1-mistral-7b - Open-Orca/Mistral-7B-OpenOrca - openchat/openchat-3.5-0106 - mlabonne/NeuralHermes-2.5-Mistral-7B - GreenNode/GreenNode-mini-7B-multilingual-v1olet - berkeley-nest/Starling-LM-7B-alpha - viethq188/LeoScorpius-7B-Chat-DPO - meta-math/MetaMath-Mistral-7B - Intel/neural-chat-7b-v3-3 inference: false model-index: - name: Moza-7B-v1.0 results: - task: type: text-generation name: Text Generation dataset: name: AI2 Reasoning Challenge (25-Shot) type: ai2_arc config: ARC-Challenge split: test args: num_few_shot: 25 metrics: - type: acc_norm value: 66.55 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: HellaSwag (10-Shot) type: hellaswag split: validation args: num_few_shot: 10 metrics: - type: acc_norm value: 83.45 name: normalized accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: MMLU (5-Shot) type: cais/mmlu config: all split: test args: num_few_shot: 5 metrics: - type: acc value: 62.77 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: TruthfulQA (0-shot) type: truthful_qa config: multiple_choice split: validation args: num_few_shot: 0 metrics: - type: mc2 value: 65.16 source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: Winogrande (5-shot) type: winogrande config: winogrande_xl split: validation args: num_few_shot: 5 metrics: - type: acc value: 77.51 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard - task: type: text-generation name: Text Generation dataset: name: GSM8k (5-shot) type: gsm8k config: main split: test args: num_few_shot: 5 metrics: - type: acc value: 62.55 name: accuracy source: url: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard?query=kidyu/Moza-7B-v1.0 name: Open LLM Leaderboard --- # Moza-7B-v1.0 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/63474d73511cd17d2c790ed7/e7hw2xIzfpUseCFEOINg7.png) This is a [meme-merge](https://en.wikipedia.org/wiki/Joke) of pre-trained language models, created using [mergekit](https://github.com/cg123/mergekit). Use at your own risk. ## Details ### Quantized Model - [GGUF](https://huggingface.co/kidyu/Moza-7B-v1.0-GGUF) ### Merge Method This model was merged using the [DARE](https://arxiv.org/abs/2311.03099) [TIES](https://arxiv.org/abs/2306.01708) merge method, using [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) as a base. The value for `density` are from [this blogpost](https://huggingface.co/blog/mlabonne/merge-models), and the weight was randomly generated and then assigned to the models, with priority (of using the bigger weight) to `NeuralHermes`, `OpenOrca`, and `neural-chat`. The models themselves are chosen by "vibes". ### Models Merged The following models were included in the merge: * [cognitivecomputations/dolphin-2.2.1-mistral-7b](https://huggingface.co/cognitivecomputations/dolphin-2.2.1-mistral-7b) * [Open-Orca/Mistral-7B-OpenOrca](https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca) * [openchat/openchat-3.5-0106](https://huggingface.co/openchat/openchat-3.5-0106) * [mlabonne/NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B) * [GreenNode/GreenNode-mini-7B-multilingual-v1olet](https://huggingface.co/GreenNode/GreenNode-mini-7B-multilingual-v1olet) * [berkeley-nest/Starling-LM-7B-alpha](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) * [viethq188/LeoScorpius-7B-Chat-DPO](https://huggingface.co/viethq188/LeoScorpius-7B-Chat-DPO) * [meta-math/MetaMath-Mistral-7B](https://huggingface.co/meta-math/MetaMath-Mistral-7B) * [Intel/neural-chat-7b-v3-3](https://huggingface.co/Intel/neural-chat-7b-v3-3) ### Prompt Format You can use `Alpaca` formatting for inference ``` ### Instruction: ### Response: ``` ### Configuration The following YAML configuration was used to produce this model: ```yaml base_model: mistralai/Mistral-7B-v0.1 models: - model: mlabonne/NeuralHermes-2.5-Mistral-7B parameters: density: 0.63 weight: 0.83 - model: Intel/neural-chat-7b-v3-3 parameters: density: 0.63 weight: 0.74 - model: meta-math/MetaMath-Mistral-7B parameters: density: 0.63 weight: 0.22 - model: openchat/openchat-3.5-0106 parameters: density: 0.63 weight: 0.37 - model: Open-Orca/Mistral-7B-OpenOrca parameters: density: 0.63 weight: 0.76 - model: cognitivecomputations/dolphin-2.2.1-mistral-7b parameters: density: 0.63 weight: 0.69 - model: viethq188/LeoScorpius-7B-Chat-DPO parameters: density: 0.63 weight: 0.38 - model: GreenNode/GreenNode-mini-7B-multilingual-v1olet parameters: density: 0.63 weight: 0.13 - model: berkeley-nest/Starling-LM-7B-alpha parameters: density: 0.63 weight: 0.33 merge_method: dare_ties parameters: normalize: true int8_mask: true dtype: bfloat16 ``` # [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard) Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_kidyu__Moza-7B-v1.0) | Metric |Value| |---------------------------------|----:| |Avg. |69.66| |AI2 Reasoning Challenge (25-Shot)|66.55| |HellaSwag (10-Shot) |83.45| |MMLU (5-Shot) |62.77| |TruthfulQA (0-shot) |65.16| |Winogrande (5-shot) |77.51| |GSM8k (5-shot) |62.55|