Training in progress, step 12375

README.md

@@ -0,0 +1,69 @@
+---
+license: mit
+base_model: gpt2
+tags:
+- generated_from_trainer
+model-index:
+- name: distily_bench_gpt2_attn
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# distily_bench_gpt2_attn
+
+This model is a fine-tuned version of [gpt2](https://huggingface.co/gpt2) on an unknown dataset.
+It achieves the following results on the evaluation set:
+- Loss: 1.1698
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 4e-05
+- train_batch_size: 8
+- eval_batch_size: 8
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: constant
+- num_epochs: 1.0
+
+### Training results
+
+| Training Loss | Epoch  | Step  | Validation Loss |
+|:-------------:|:------:|:-----:|:---------------:|
+| No log        | 0      | 0     | 6.0237          |
+| 2.0831        | 0.0808 | 1000  | 2.0503          |
+| 1.9875        | 0.1616 | 2000  | 1.8577          |
+| 1.7222        | 0.2424 | 3000  | 1.7444          |
+| 1.625         | 0.3232 | 4000  | 1.6547          |
+| 1.5466        | 0.4040 | 5000  | 1.5704          |
+| 1.482         | 0.4848 | 6000  | 1.4916          |
+| 1.3055        | 0.5657 | 7000  | 1.4296          |
+| 1.3487        | 0.6465 | 8000  | 1.3749          |
+| 1.5284        | 0.7273 | 9000  | 1.3261          |
+| 1.3073        | 0.8081 | 10000 | 1.2775          |
+| 1.2886        | 0.8889 | 11000 | 1.2243          |
+| 1.2192        | 0.9697 | 12000 | 1.1816          |
+
+
+### Framework versions
+
+- Transformers 4.44.0
+- Pytorch 2.3.0
+- Datasets 2.20.0
+- Tokenizers 0.19.1

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.44.0"
+}

logs/attn_loss_fn=mse, attn_weight=2.0/events.out.tfevents.1723656416.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d63d52043af7b1d5479328d440a06d9caeea4a3e3cfbee8b564e81e2d647c348
+size 3345958

logs/events.out.tfevents.1723656174.93d6cbb3ad53

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab348a72b30d9979f79bc613e8a2a8cb80fe606b94154ebebd14b562678c9e96
+size 520

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:752e08ae803f875787090f2660ae0f29c3e865497c67e4559fac7dfbc442cb55
+oid sha256:6a0ec1453377d7bf4e231beb11459860f4cbd412ad0726105d10c4786ca76a6c
 size 248894656

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:09e81d64ef5941f016615495092673584a6bf2022c8c7140fdc6229116fd782b
-size 907106500
+oid sha256:80d3e5ed917d613007ba4a4ebff101a440047d550e395dc35044c6d2709142e0
+size 907106564