phi3.5-pro-10-08

This model is a fine-tuned version of E:\mergekit\phi3.5-pro on the Magpie_Qwen2_Pro_300k and the Magpie_Llama_3.1_Pro_MT_300k datasets. It achieves the following results on the evaluation set:

• Loss: 0.8538

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0001
• train_batch_size: 2
• eval_batch_size: 2
• seed: 42
• gradient_accumulation_steps: 8
• total_train_batch_size: 16
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• training_steps: 1000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.7784	0.0080	100	0.8538
0.8605	0.0160	200	0.8538
0.8386	0.0240	300	0.8538
0.8201	0.0320	400	0.8538
0.8661	0.0400	500	0.8538
0.776	0.0480	600	0.8538
0.8377	0.0561	700	0.8538
0.8541	0.0641	800	0.8538
0.799	0.0721	900	0.8538
0.8146	0.0801	1000	0.8538

Framework versions

• Transformers 4.45.0
• Pytorch 2.4.0+cu124
• Datasets 2.21.0
• Tokenizers 0.20.0