Hugging Face's logo

pszemraj
/
mega-ar-350m-v0.13

Text Generation
Transformers
Safetensors
English
mega
Generated from Trainer
Inference Endpoints
Model card Files Community
pszemraj commited on
Commit
f473ac4
•
1 Parent(s): 7a12219

Model save

Browse files
Files changed (3) hide show
  1. README.md +88 -0
  2. generation_config.json +12 -12
  3. model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,88 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: pszemraj/mega-ar-350m-v0.12-napierone_epub
4
+ tags:
5
+ - generated_from_trainer
6
+ metrics:
7
+ - accuracy
8
+ model-index:
9
+ - name: mega-ar-350m-v0.12-napierone_epub-UltraTextbooks-2.1-fw_mix-vN
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ # mega-ar-350m-v0.12-napierone_epub-UltraTextbooks-2.1-fw_mix-vN
17
+
18
+ This model is a fine-tuned version of [pszemraj/mega-ar-350m-v0.12-napierone_epub](https://huggingface.co/pszemraj/mega-ar-350m-v0.12-napierone_epub) on an unknown dataset.
19
+ It achieves the following results on the evaluation set:
20
+ - Loss: 1.9926
21
+ - Accuracy: 0.5884
22
+ - Num Input Tokens Seen: 3460300800
23
+
24
+ ## Model description
25
+
26
+ More information needed
27
+
28
+ ## Intended uses & limitations
29
+
30
+ More information needed
31
+
32
+ ## Training and evaluation data
33
+
34
+ More information needed
35
+
36
+ ## Training procedure
37
+
38
+ ### Training hyperparameters
39
+
40
+ The following hyperparameters were used during training:
41
+ - learning_rate: 5e-05
42
+ - train_batch_size: 1
43
+ - eval_batch_size: 1
44
+ - seed: 80085
45
+ - distributed_type: multi-GPU
46
+ - num_devices: 3
47
+ - gradient_accumulation_steps: 32
48
+ - total_train_batch_size: 96
49
+ - total_eval_batch_size: 3
50
+ - optimizer: Adam with betas=(0.9,0.985) and epsilon=1e-08
51
+ - lr_scheduler_type: linear
52
+ - lr_scheduler_warmup_ratio: 0.05
53
+ - num_epochs: 1.0
54
+
55
+ ### Training results
56
+
57
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy | Input Tokens Seen |
58
+ |:-------------:|:------:|:----:|:---------------:|:--------:|:-----------------:|
59
+ | 2.2374 | 0.0454 | 400 | 2.1871 | 0.5588 | 157286400 |
60
+ | 2.143 | 0.0907 | 800 | 2.1336 | 0.5665 | 314572800 |
61
+ | 2.1272 | 0.1361 | 1200 | 2.1092 | 0.5698 | 471859200 |
62
+ | 2.1243 | 0.1814 | 1600 | 2.0929 | 0.5725 | 629145600 |
63
+ | 2.1021 | 0.2268 | 2000 | 2.0794 | 0.5747 | 786432000 |
64
+ | 2.0794 | 0.2721 | 2400 | 2.0687 | 0.5762 | 943718400 |
65
+ | 2.0843 | 0.3175 | 2800 | 2.0592 | 0.5776 | 1101004800 |
66
+ | 2.0571 | 0.3628 | 3200 | 2.0507 | 0.5793 | 1258291200 |
67
+ | 2.0841 | 0.4082 | 3600 | 2.0435 | 0.5802 | 1415577600 |
68
+ | 2.0484 | 0.4535 | 4000 | 2.0363 | 0.5813 | 1572864000 |
69
+ | 2.0199 | 0.4989 | 4400 | 2.0315 | 0.5820 | 1730150400 |
70
+ | 2.0361 | 0.5442 | 4800 | 2.0261 | 0.5829 | 1887436800 |
71
+ | 2.057 | 0.5896 | 5200 | 2.0207 | 0.5838 | 2044723200 |
72
+ | 2.0234 | 0.6349 | 5600 | 2.0163 | 0.5845 | 2202009600 |
73
+ | 2.073 | 0.6803 | 6000 | 2.0120 | 0.5850 | 2359296000 |
74
+ | 2.058 | 0.7256 | 6400 | 2.0074 | 0.5862 | 2516582400 |
75
+ | 2.0253 | 0.7710 | 6800 | 2.0041 | 0.5866 | 2673868800 |
76
+ | 1.995 | 0.8163 | 7200 | 2.0010 | 0.5872 | 2831155200 |
77
+ | 1.9735 | 0.8617 | 7600 | 1.9987 | 0.5875 | 2988441600 |
78
+ | 1.9799 | 0.9070 | 8000 | 1.9960 | 0.5880 | 3145728000 |
79
+ | 2.0056 | 0.9524 | 8400 | 1.9942 | 0.5882 | 3303014400 |
80
+ | 1.9961 | 0.9977 | 8800 | 1.9926 | 0.5884 | 3460300800 |
81
+
82
+
83
+ ### Framework versions
84
+
85
+ - Transformers 4.40.2
86
+ - Pytorch 2.2.0+cu121
87
+ - Datasets 2.19.1
88
+ - Tokenizers 0.19.1
generation_config.json CHANGED
@@ -1,13 +1,13 @@
1
  {
2
- "_from_model_config":true,
3
- "bos_token_id":128000,
4
- "eos_token_id":128001,
5
- "max_new_tokens":64,
6
- "do_sample":true,
7
- "temperature":0.6,
8
- "repetition_penalty":1.20,
9
- "no_repeat_ngram_size":4,
10
- "eta_cutoff":0.0006,
11
- "renormalize_logits":true,
12
- "transformers_version":"4.40.1"
13
- }
 
1
  {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 128000,
4
+ "do_sample": true,
5
+ "eos_token_id": 128001,
6
+ "epsilon_cutoff": 0.0003,
7
+ "max_new_tokens": 64,
8
+ "no_repeat_ngram_size": 4,
9
+ "renormalize_logits": true,
10
+ "repetition_penalty": 1.15,
11
+ "temperature": 0.7,
12
+ "transformers_version": "4.40.2"
13
+ }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:acba165f9aa49e64fd82c3bba1cc3f7ef1336fd5abba02a3851ac45bea13a269
3
  size 1398219896
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d4e3e06c10184937e36dc4907efc96f6c28d9723f136763f6202730f2af89ef2
3
  size 1398219896