PygTesting/sum_qlora_4pochs

See axolotl config

axolotl version: 0.4.1

base_model: mistralai/Mistral-Nemo-Base-2407
model_type: AutoModelForCausalLM
tokenizer_type: AutoTokenizer

load_in_8bit: false
load_in_4bit: true
strict: false

plugins:
  - axolotl.integrations.liger.LigerPlugin
liger_rope: true
liger_rms_norm: true
liger_swiglu: true
liger_fused_linear_cross_entropy: true

adapter: qlora
lora_r: 32
lora_alpha: 16
lora_dropout: 0.05
lora_target_linear: true
lora_fan_in_fan_out:

datasets:
  - path: /home/austin/disk1/summaries_fixed.jsonl
    type: sharegpt
dataset_prepared_path: last_run_prepared
val_set_size: 0.01
output_dir: ./qlora_outputs

sequence_len: 8192
sample_packing: true
eval_sample_packing: false
pad_to_sequence_len: true

wandb_project: summarization-qlora
wandb_entity:
wandb_watch:
wandb_name: actual_run1
wandb_log_model:
 
 #unsloth_cross_entropy_loss: true

gradient_accumulation_steps: 1
micro_batch_size: 1
num_epochs: 4
optimizer: adamw_bnb_8bit
lr_scheduler: cosine
learning_rate: 0.0002

train_on_inputs: false
group_by_length: false
bf16: auto
fp16:
tf32: false

gradient_checkpointing: true
gradient_checkpointing_kwargs:
  use_reentrant: false
early_stopping_patience:
resume_from_checkpoint:
logging_steps: 1
xformers_attention: false
flash_attention: true

loss_watchdog_threshold: 5.0
loss_watchdog_patience: 3

warmup_steps: 25
evals_per_epoch: 4
eval_table_size:
saves_per_epoch: 4
debug:
deepspeed: ./deepspeed_configs/zero2.json
weight_decay: 0.0
fsdp:
  #  - full_shard
  #  - auto_wrap
fsdp_config:
  #  fsdp_limit_all_gathers: true
  #  fsdp_activation_checkpointing: true
  #  fsdp_sync_module_states: true
  #  fsdp_offload_params: false
  #  fsdp_use_orig_params: false
  #  fsdp_cpu_ram_efficient_loading: false
  #  fsdp_transformer_layer_cls_to_wrap: MistralDecoderLayer
  #  fsdp_state_dict_type: FULL_STATE_DICT
  #  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
special_tokens:
  pad_token: </s>

qlora_outputs

This model is a fine-tuned version of mistralai/Mistral-Nemo-Base-2407 on the None dataset. It achieves the following results on the evaluation set:

• Loss: 1.5617

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 1
• eval_batch_size: 1
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• total_train_batch_size: 4
• total_eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• lr_scheduler_warmup_steps: 25
• num_epochs: 4

Training results

Training Loss	Epoch	Step	Validation Loss
2.0177	0.0014	1	1.6514
1.6259	0.2507	177	1.2032
1.4232	0.5014	354	1.1897
1.6835	0.7521	531	1.1985
1.6514	1.0028	708	1.1874
1.4538	1.2365	885	1.2166
1.2421	1.4873	1062	1.2224
1.2844	1.7380	1239	1.2330
1.4152	1.9887	1416	1.2345
1.1668	2.2252	1593	1.3476
1.1249	2.4759	1770	1.3608
0.921	2.7266	1947	1.3793
0.7824	2.9773	2124	1.3906
1.1759	3.2040	2301	1.5438
0.6625	3.4547	2478	1.5644
0.8959	3.7054	2655	1.5617

Framework versions

• PEFT 0.12.0
• Transformers 4.44.0
• Pytorch 2.3.0+cu121
• Datasets 2.20.0
• Tokenizers 0.19.1