mgv99 commited on
Commit
f69bde1
1 Parent(s): fdc78ee

Upload folder using huggingface_hub

Browse files
Files changed (2) hide show
  1. README.md +65 -0
  2. config.json +32 -0
README.md ADDED
@@ -0,0 +1,65 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model:
3
+ - openchat/openchat-3.5-1210
4
+ - beowolx/CodeNinja-1.0-OpenChat-7B
5
+ license: mit
6
+ tags:
7
+ - moe
8
+ - frankenmoe
9
+ - merge
10
+ - mergekit
11
+ - openchat/openchat-3.5-1210
12
+ - beowolx/CodeNinja-1.0-OpenChat-7B
13
+ ---
14
+
15
+ # prueba-moe
16
+
17
+ prueba-moe is a Mixture of Experts (MoE) made with the following models using [Mergekit](https://github.com/arcee-ai/mergekit):
18
+ * [openchat/openchat-3.5-1210](https://huggingface.co/openchat/openchat-3.5-1210)
19
+ * [beowolx/CodeNinja-1.0-OpenChat-7B](https://huggingface.co/beowolx/CodeNinja-1.0-OpenChat-7B)
20
+
21
+ ## 🧩 Configuration
22
+
23
+ ```yamlbase_model: mlabonne/Marcoro14-7B-slerp
24
+ experts:
25
+ - positive_prompts:
26
+ - chat
27
+ - assistant
28
+ - tell me
29
+ - explain
30
+ - what is
31
+ source_model: openchat/openchat-3.5-1210
32
+ - positive_prompts:
33
+ - code
34
+ - python
35
+ - javascript
36
+ - programming
37
+ - algorithm
38
+ source_model: beowolx/CodeNinja-1.0-OpenChat-7B
39
+ experts_per_token: 2
40
+ gate_mode: cheap_embed
41
+ ```
42
+
43
+ ## 💻 Usage
44
+
45
+ ```python
46
+ !pip install -qU transformers bitsandbytes accelerate
47
+
48
+ from transformers import AutoTokenizer
49
+ import transformers
50
+ import torch
51
+
52
+ model = "mgv99/prueba-moe"
53
+
54
+ tokenizer = AutoTokenizer.from_pretrained(model)
55
+ pipeline = transformers.pipeline(
56
+ "text-generation",
57
+ model=model,
58
+ model_kwargs={"torch_dtype": torch.float16, "load_in_4bit": True},
59
+ )
60
+
61
+ messages = [{"role": "user", "content": "Explain what a Mixture of Experts is in less than 100 words."}]
62
+ prompt = pipeline.tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
63
+ outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
64
+ print(outputs[0]["generated_text"])
65
+ ```
config.json ADDED
@@ -0,0 +1,32 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_name_or_path": "mlabonne/Marcoro14-7B-slerp",
3
+ "architectures": [
4
+ "MixtralForCausalLM"
5
+ ],
6
+ "attention_dropout": 0.0,
7
+ "bos_token_id": 1,
8
+ "eos_token_id": 2,
9
+ "head_dim": 128,
10
+ "hidden_act": "silu",
11
+ "hidden_size": 4096,
12
+ "initializer_range": 0.02,
13
+ "intermediate_size": 14336,
14
+ "max_position_embeddings": 32768,
15
+ "model_type": "mixtral",
16
+ "num_attention_heads": 32,
17
+ "num_experts_per_tok": 2,
18
+ "num_hidden_layers": 32,
19
+ "num_key_value_heads": 8,
20
+ "num_local_experts": 2,
21
+ "output_router_logits": false,
22
+ "rms_norm_eps": 1e-05,
23
+ "rope_theta": 10000.0,
24
+ "router_aux_loss_coef": 0.001,
25
+ "router_jitter_noise": 0.0,
26
+ "sliding_window": null,
27
+ "tie_word_embeddings": false,
28
+ "torch_dtype": "bfloat16",
29
+ "transformers_version": "4.44.2",
30
+ "use_cache": true,
31
+ "vocab_size": 32000
32
+ }