README.md

metadata

base_model:
  - flammenai/Mahou-1.3a-mistral-7B
  - flammenai/Mahou-1.2a-mistral-7B
library_name: transformers
tags:
  - mergekit
  - merge

Mahou-1.3-M2-mistral-7B

Mahou is designed to provide short messages in a conversational context. It is capable of casual conversation and character roleplay.

Chat Format

This model has been trained to use ChatML format. Note the additional tokens in tokenizer_config.json.

<|im_start|>system
{{system}}<|im_end|>
<|im_start|>{{char}}
{{message}}<|im_end|>
<|im_start|>{{user}}
{{message}}<|im_end|>

Roleplay Format

• Speech without quotes.
• Actions in *asterisks*

*leans against wall cooly* so like, i just casted a super strong spell at magician academy today, not gonna lie, felt badass.

ST Settings

1. Use ChatML for the Context Template.
2. Enable Instruct Mode.
3. Use the Mahou preset.
4. Recommended: Add newline as a stopping string: ["\n"]

Method

The following YAML configuration was used to produce this model using mergekit:

models:
  - model: flammenai/Mahou-1.3a-mistral-7B
    layer_range: [0, 32]
  - model: flammenai/Mahou-1.2a-mistral-7B
    layer_range: [0, 32]
merge_method: slerp
base_model: flammenai/Mahou-1.3a-mistral-7B
tokenizer_source: 'union'
parameters:
  embed_slerp: true
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0]
    - value: 0.5
dtype: bfloat16