See axolotl config

axolotl version: 0.4.1

base_model: microsoft/Phi-3.5-mini-instruct
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer
chat_template: phi_3

load_in_8bit: false
load_in_4bit: false
strict: false

datasets:
  - path: flydust/CodeGen_50000_Test_Full
    type: chat_template
    field_messages: conversations
    # The key in the message turn that contains the role. Default is "role".
    message_field_role: from
    # The key in the message turn that contains the content. Default is "content".
    message_field_content: value
    # Optional[Dict[str, List]]. Roles mapping for the messages.
    roles:
      user: ["human", "user"]
      assistant: ["gpt", "assistant", "ai"]
      system: ["system"]


dataset_prepared_path: last_run_prepared
val_set_size: 0.001
output_dir: axolotl_out/Phi-3.5-mini-instruct-Code50000-Test_Full

sequence_len: 4096
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project: SynDa
wandb_entity:
wandb_watch:
wandb_name: Phi-3.5-mini-instruct-Code50000-Test_Full
wandb_log_model:
hub_model_id: flydust/Phi-3.5-mini-instruct-Code50000-Test_Full

gradient_accumulation_steps: 4
micro_batch_size: 1
num_epochs: 2
optimizer: paged_adamw_8bit
lr_scheduler: cosine
learning_rate: 2e-5

train_on_inputs: false
group_by_length: false
bf16: true
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention:
# Disable flash attention
flash_attention: true
# sdp_attention: falses
# eager_attention: true

warmup_ratio: 0.1
evals_per_epoch: 10
eval_table_size:
saves_per_epoch: 1
debug:
deepspeed:
weight_decay: 0.0
fsdp:
fsdp_config:
special_tokens:

Phi-3.5-mini-instruct-Code50000-Test_Full

This model is a fine-tuned version of microsoft/Phi-3.5-mini-instruct on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.1916

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 39
• num_epochs: 2

Training results

Training Loss	Epoch	Step	Validation Loss
0.5151	0.0050	1	0.6127
0.2421	0.1001	20	0.2490
0.2409	0.2003	40	0.2145
0.1907	0.3004	60	0.2072
0.2105	0.4005	80	0.2021
0.2105	0.5006	100	0.1995
0.2001	0.6008	120	0.1964
0.1993	0.7009	140	0.1964
0.1936	0.8010	160	0.1952
0.19	0.9011	180	0.1941
0.197	1.0013	200	0.1931
0.1925	1.0987	220	0.1920
0.1933	1.1987	240	0.1923
0.2131	1.2988	260	0.1915
0.1972	1.3987	280	0.1912
0.1939	1.4988	300	0.1911
0.1905	1.5987	320	0.1913
0.1751	1.6987	340	0.1910
0.1797	1.7988	360	0.1923
0.2036	1.8988	380	0.1916

Framework versions

• Transformers 4.45.2
• Pytorch 2.5.1+cu124
• Datasets 3.0.1
• Tokenizers 0.20.3