Summary
Distilled with Distily library using teacher model gpt2 on dataset wikimedia/wikipedia.
Model Architecture:
- Architecture:
GPT2LMHeadModel - Total Parameters: 124,439,808
- Data Type (dtype): torch.bfloat16
- Model Size: 0.24 GB
Benchmark Metrics Comparison
| Metric | | | :--- |
Resource Usage Comparison
- VRAM Use: 7.4923 GB
Distillation (Teacher -> Student) Architecture Difference:
- Architecture:
GPT2LMHeadModel->GPT2LMHeadModel - Total Parameters: 124,439,808 -> 124,439,808
- Data Type (dtype): torch.bfloat16 -> torch.bfloat16
- Model Size: 0.24 GB -> 0.24 GB
Module Diff Details
Train Dataset
Trained on 3,223,855 tokens from the wikimedia/wikipedia dataset.
- Num Samples:
3,960 - Subset:
20231101.en - Split:
train
Training Objective
DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl))
Hyperparameters
The following hyperparameters were used during training:
Expand
- learning_rate:
0.0001 - train_batch_size:
4 - eval_batch_size:
8 - seed:
42 - optimizer:
Adam with betas=(0.9,0.999) and epsilon=1e-08 - lr_scheduler_type:
constant - lr_scheduler_warmup_ratio:
0.2 - num_epochs:
1.0 - distillation_objective:
DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl)) - train_embeddings:
True - lr_scheduler:
<torch.optim.lr_scheduler.LambdaLR object at 0x7fd25af72440> - student_model_name_or_path:
None - student_config_name_or_path:
None - student_model_config:
None - reinitialize_weights:
None - copy_teacher_modules:
[('lm_head', False)] - student_model_as_bitnet:
False - dropout:
None - teacher_model_name_or_path:
gpt2 - teacher_load_in_8bit:
False - teacher_load_in_4bit:
False - dataset_uri:
wikimedia/wikipedia - dataset_subset:
20231101.en - dataset_split:
train - dataset_column_name:
text - dataset_sample_size:
4000 - dataset_test_size:
0.01 - gradient_accumulation_steps:
1 - weight_decay:
0.0 - max_grad_norm:
1.0 - warmup_ratio:
0.2 - warmup_steps:
0 - gradient_checkpointing:
True
Framework Versions
- Distily 0.3.0
- Transformers 4.44.2
- Pytorch 2.3.0
- Datasets 2.21.0
- Downloads last month
- 2
Model tree for distily/distily_verify_compile
Base model
openai-community/gpt2