BirdAST_Baseline_GroupKFold_training.log

################################### TRAIN_CONFIG ###################################
dataset_dir: ./Audio_XenoCanto
labels_list: ./xeno_labels.csv
model_name: BirdAST_Baseline_GroupKFold
backbone_name: MIT/ast-finetuned-audioset-10-10-0.4593
n_classes: 728
audio_sr: 16000
segment_length: 10
fft_window: 0.025
hop_window_length: 0.01
n_mels: 128
low_cut: 1000
high_cut: 8000
top_db: 100
batch_size: 16
num_workers: 0
n_splits: 5
log_dir: ./training_logs
max_lr: 1e-05
epochs: 10
weight_decay: 0.01
lr_final_div: 1000
amp: True
grad_accum_steps: 1
max_grad_norm: 10000000.0
print_epoch_freq: 1
print_freq: 500
random_seed: 2046
copy: <classmethod(<function Config.copy at 0x7b4f57baf1c0>)>
################################################################################
Failed to detect the name of this notebook, you can set it manually with the WANDB_NOTEBOOK_NAME environment variable to enable code saving.
Epoch 1 [0/559] | Train Loss: 0.3797 Grad: 132458.4531 LR: 4.0008e-07 | Elapse: 5.22s
Epoch 1 [500/559] | Train Loss: 0.1767 Grad: 17217.5918 LR: 9.7549e-06 | Elapse: 632.27s
Epoch 1 [558/559] | Train Loss: 0.1659 Grad: 38565.3086 LR: 1.0000e-05 | Elapse: 704.92s
Epoch 1 [0/140] | Valid Loss: 0.0956 | Elapse: 1.77s
Epoch 1 [139/140] | Valid Loss: 0.1626 | Elapse: 179.13s
Epoch 1 - Train Loss: 0.1659 - Valid Loss: 0.5170 - Elapsed Time: 902.38s
- Epoch 1: Best model found with loss = 0.5170.
Epoch 2 [0/559] | Train Loss: 0.3837 Grad: 82366.4531 LR: 1.0000e-05 | Elapse: 1.39s
Epoch 2 [500/559] | Train Loss: 0.1670 Grad: 26346.2246 LR: 9.7564e-06 | Elapse: 647.59s
Epoch 2 [558/559] | Train Loss: 0.1563 Grad: 53784.7227 LR: 9.6974e-06 | Elapse: 716.52s
Epoch 2 [0/140] | Valid Loss: 0.0949 | Elapse: 1.36s
Epoch 2 [139/140] | Valid Loss: 0.1759 | Elapse: 176.02s
Epoch 2 - Train Loss: 0.1563 - Valid Loss: 0.5562 - Elapsed Time: 910.59s
- Epoch 2: Best model found with loss = 0.5562.
Epoch 3 [0/559] | Train Loss: 0.3296 Grad: 136677.4531 LR: 9.6963e-06 | Elapse: 1.60s
Epoch 3 [500/559] | Train Loss: 0.1347 Grad: 29127.7148 LR: 8.9422e-06 | Elapse: 630.69s
Epoch 3 [558/559] | Train Loss: 0.1259 Grad: 57361.0430 LR: 8.8283e-06 | Elapse: 700.52s
Epoch 3 [0/140] | Valid Loss: 0.0909 | Elapse: 1.56s
Epoch 3 [139/140] | Valid Loss: 0.1843 | Elapse: 176.22s
Epoch 3 - Train Loss: 0.1259 - Valid Loss: 0.6019 - Elapsed Time: 894.87s
- Epoch 3: Best model found with loss = 0.6019.
Epoch 4 [0/559] | Train Loss: 0.2495 Grad: 174822.3438 LR: 8.8263e-06 | Elapse: 1.03s
Epoch 4 [500/559] | Train Loss: 0.0971 Grad: 30384.9941 LR: 7.6526e-06 | Elapse: 616.92s
Epoch 4 [558/559] | Train Loss: 0.0909 Grad: 54755.8555 LR: 7.4974e-06 | Elapse: 686.08s
Epoch 4 [0/140] | Valid Loss: 0.0883 | Elapse: 0.96s
Epoch 4 [139/140] | Valid Loss: 0.1906 | Elapse: 170.98s
Epoch 4 - Train Loss: 0.0909 - Valid Loss: 0.6292 - Elapsed Time: 875.26s
- Epoch 4: Best model found with loss = 0.6292.
Epoch 5 [0/559] | Train Loss: 0.1445 Grad: 179717.0781 LR: 7.4947e-06 | Elapse: 1.67s
Epoch 5 [500/559] | Train Loss: 0.0679 Grad: 31367.4883 LR: 6.0431e-06 | Elapse: 636.79s
Epoch 5 [558/559] | Train Loss: 0.0638 Grad: 46204.8477 LR: 5.8653e-06 | Elapse: 710.08s
Epoch 5 [0/140] | Valid Loss: 0.0862 | Elapse: 1.37s
Epoch 5 [139/140] | Valid Loss: 0.1974 | Elapse: 172.42s
Epoch 5 - Train Loss: 0.0638 - Valid Loss: 0.6417 - Elapsed Time: 900.70s
- Epoch 5: Best model found with loss = 0.6417.
Epoch 6 [0/559] | Train Loss: 0.0752 Grad: 150651.5312 LR: 5.8623e-06 | Elapse: 1.26s
Epoch 6 [500/559] | Train Loss: 0.0498 Grad: 30212.4238 LR: 4.3078e-06 | Elapse: 625.35s
Epoch 6 [558/559] | Train Loss: 0.0471 Grad: 45234.8984 LR: 4.1289e-06 | Elapse: 698.58s
Epoch 6 [0/140] | Valid Loss: 0.0843 | Elapse: 1.56s
Epoch 6 [139/140] | Valid Loss: 0.2014 | Elapse: 168.62s
Epoch 6 - Train Loss: 0.0471 - Valid Loss: 0.6506 - Elapsed Time: 885.11s
- Epoch 6: Best model found with loss = 0.6506.
Epoch 7 [0/559] | Train Loss: 0.0401 Grad: 110378.2734 LR: 4.1258e-06 | Elapse: 1.55s
Epoch 7 [500/559] | Train Loss: 0.0401 Grad: 29949.4160 LR: 2.6560e-06 | Elapse: 747.46s
Epoch 7 [558/559] | Train Loss: 0.0381 Grad: 47635.7148 LR: 2.4976e-06 | Elapse: 850.90s
Epoch 7 [0/140] | Valid Loss: 0.0835 | Elapse: 1.84s
Epoch 7 [139/140] | Valid Loss: 0.2044 | Elapse: 247.83s
Epoch 7 - Train Loss: 0.0381 - Valid Loss: 0.6516 - Elapsed Time: 1122.23s
- Epoch 7: Best model found with loss = 0.6516.
Epoch 8 [0/559] | Train Loss: 0.0310 Grad: 93998.0625 LR: 2.4949e-06 | Elapse: 2.01s
Epoch 8 [500/559] | Train Loss: 0.0364 Grad: 34944.8828 LR: 1.2869e-06 | Elapse: 898.62s
Epoch 8 [558/559] | Train Loss: 0.0347 Grad: 48920.4258 LR: 1.1681e-06 | Elapse: 1001.16s
Epoch 8 [0/140] | Valid Loss: 0.0855 | Elapse: 1.69s
Epoch 8 [139/140] | Valid Loss: 0.2072 | Elapse: 250.89s
Epoch 8 - Train Loss: 0.0347 - Valid Loss: 0.6495 - Elapsed Time: 1275.92s
Epoch 9 [0/559] | Train Loss: 0.0334 Grad: 111821.3047 LR: 1.1661e-06 | Elapse: 1.79s
Epoch 9 [500/559] | Train Loss: 0.0380 Grad: 48075.5664 LR: 3.6575e-07 | Elapse: 896.79s
Epoch 9 [558/559] | Train Loss: 0.0362 Grad: 48004.2852 LR: 3.0086e-07 | Elapse: 999.50s
Epoch 9 [0/140] | Valid Loss: 0.0802 | Elapse: 1.68s
Epoch 9 [139/140] | Valid Loss: 0.2040 | Elapse: 247.83s
Epoch 9 - Train Loss: 0.0362 - Valid Loss: 0.6773 - Elapsed Time: 1272.24s
- Epoch 9: Best model found with loss = 0.6773.
Epoch 10 [0/559] | Train Loss: 0.0419 Grad: 138725.0625 LR: 2.9979e-07 | Elapse: 1.85s
Epoch 10 [500/559] | Train Loss: 0.0442 Grad: 51908.7266 LR: 3.5668e-09 | Elapse: 851.64s
Epoch 10 [558/559] | Train Loss: 0.0418 Grad: 36428.0664 LR: 4.0097e-10 | Elapse: 950.39s
Epoch 10 [0/140] | Valid Loss: 0.0763 | Elapse: 1.74s
Epoch 10 [139/140] | Valid Loss: 0.2015 | Elapse: 253.39s
Epoch 10 - Train Loss: 0.0418 - Valid Loss: 0.6896 - Elapsed Time: 1228.92s
- Epoch 10: Best model found with loss = 0.6896.
Fold 0 | Time: 171.93min | Overall Evaluation Loss: 0.6896
Epoch 1 [0/559] | Train Loss: 0.4015 Grad: 130138.6250 LR: 4.0008e-07 | Elapse: 1.81s
Epoch 1 [500/559] | Train Loss: 0.1759 Grad: 863.0330 LR: 9.7549e-06 | Elapse: 869.10s
Epoch 1 [558/559] | Train Loss: 0.1663 Grad: 33445.6641 LR: 1.0000e-05 | Elapse: 956.09s
Epoch 1 [0/140] | Valid Loss: 0.2185 | Elapse: 1.43s
Epoch 1 [139/140] | Valid Loss: 0.1571 | Elapse: 206.29s
Epoch 1 - Train Loss: 0.1663 - Valid Loss: 0.5072 - Elapsed Time: 1181.07s
- Epoch 1: Best model found with loss = 0.5072.
Epoch 2 [0/559] | Train Loss: 0.3793 Grad: 81459.7891 LR: 1.0000e-05 | Elapse: 1.45s
Epoch 2 [500/559] | Train Loss: 0.1659 Grad: 1246.6095 LR: 9.7564e-06 | Elapse: 724.53s
Epoch 2 [558/559] | Train Loss: 0.1560 Grad: 45349.8438 LR: 9.6974e-06 | Elapse: 796.86s
Epoch 2 [0/140] | Valid Loss: 0.2406 | Elapse: 1.39s
Epoch 2 [139/140] | Valid Loss: 0.1642 | Elapse: 172.62s
Epoch 2 - Train Loss: 0.1560 - Valid Loss: 0.5597 - Elapsed Time: 988.42s
- Epoch 2: Best model found with loss = 0.5597.
Epoch 3 [0/559] | Train Loss: 0.3372 Grad: 126511.1250 LR: 9.6963e-06 | Elapse: 1.61s
Epoch 3 [500/559] | Train Loss: 0.1332 Grad: 1709.5671 LR: 8.9422e-06 | Elapse: 626.50s
Epoch 3 [558/559] | Train Loss: 0.1245 Grad: 48516.1641 LR: 8.8283e-06 | Elapse: 698.54s
Epoch 3 [0/140] | Valid Loss: 0.2499 | Elapse: 1.15s
Epoch 3 [139/140] | Valid Loss: 0.1690 | Elapse: 175.40s
Epoch 3 - Train Loss: 0.1245 - Valid Loss: 0.5997 - Elapsed Time: 892.90s
- Epoch 3: Best model found with loss = 0.5997.
Epoch 4 [0/559] | Train Loss: 0.2329 Grad: 165485.4688 LR: 8.8263e-06 | Elapse: 1.42s
Epoch 4 [500/559] | Train Loss: 0.0928 Grad: 2085.9751 LR: 7.6526e-06 | Elapse: 617.80s
Epoch 4 [558/559] | Train Loss: 0.0867 Grad: 45565.9609 LR: 7.4974e-06 | Elapse: 690.54s
Epoch 4 [0/140] | Valid Loss: 0.2734 | Elapse: 1.55s
Epoch 4 [139/140] | Valid Loss: 0.1746 | Elapse: 167.59s
Epoch 4 - Train Loss: 0.0867 - Valid Loss: 0.6215 - Elapsed Time: 877.07s
- Epoch 4: Best model found with loss = 0.6215.
Epoch 5 [0/559] | Train Loss: 0.1356 Grad: 175726.2500 LR: 7.4947e-06 | Elapse: 1.12s
Epoch 5 [500/559] | Train Loss: 0.0635 Grad: 2302.8323 LR: 6.0431e-06 | Elapse: 619.62s
Epoch 5 [558/559] | Train Loss: 0.0595 Grad: 41125.3477 LR: 5.8653e-06 | Elapse: 690.76s
Epoch 5 [0/140] | Valid Loss: 0.3010 | Elapse: 1.17s
Epoch 5 [139/140] | Valid Loss: 0.1791 | Elapse: 169.10s
Epoch 5 - Train Loss: 0.0595 - Valid Loss: 0.6472 - Elapsed Time: 878.90s
- Epoch 5: Best model found with loss = 0.6472.
Epoch 6 [0/559] | Train Loss: 0.0700 Grad: 136908.1094 LR: 5.8623e-06 | Elapse: 1.13s
Epoch 6 [500/559] | Train Loss: 0.0446 Grad: 2514.1721 LR: 4.3078e-06 | Elapse: 625.46s
Epoch 6 [558/559] | Train Loss: 0.0420 Grad: 37248.8633 LR: 4.1289e-06 | Elapse: 697.66s
Epoch 6 [0/140] | Valid Loss: 0.3092 | Elapse: 1.25s
Epoch 6 [139/140] | Valid Loss: 0.1812 | Elapse: 171.69s
Epoch 6 - Train Loss: 0.0420 - Valid Loss: 0.6583 - Elapsed Time: 888.53s
- Epoch 6: Best model found with loss = 0.6583.
Epoch 7 [0/559] | Train Loss: 0.0358 Grad: 92237.4297 LR: 4.1258e-06 | Elapse: 1.29s
Epoch 7 [500/559] | Train Loss: 0.0349 Grad: 2724.6714 LR: 2.6560e-06 | Elapse: 625.38s
Epoch 7 [558/559] | Train Loss: 0.0330 Grad: 36025.4375 LR: 2.4976e-06 | Elapse: 692.62s
Epoch 7 [0/140] | Valid Loss: 0.3133 | Elapse: 1.35s
Epoch 7 [139/140] | Valid Loss: 0.1820 | Elapse: 169.90s
Epoch 7 - Train Loss: 0.0330 - Valid Loss: 0.6669 - Elapsed Time: 881.25s
- Epoch 7: Best model found with loss = 0.6669.
Epoch 8 [0/559] | Train Loss: 0.0239 Grad: 68634.6406 LR: 2.4949e-06 | Elapse: 0.94s
Epoch 8 [500/559] | Train Loss: 0.0310 Grad: 2664.3289 LR: 1.2869e-06 | Elapse: 623.73s
Epoch 8 [558/559] | Train Loss: 0.0293 Grad: 35448.2188 LR: 1.1681e-06 | Elapse: 693.87s
Epoch 8 [0/140] | Valid Loss: 0.3229 | Elapse: 1.65s
Epoch 8 [139/140] | Valid Loss: 0.1835 | Elapse: 170.20s
Epoch 8 - Train Loss: 0.0293 - Valid Loss: 0.6775 - Elapsed Time: 883.04s
- Epoch 8: Best model found with loss = 0.6775.
Epoch 9 [0/559] | Train Loss: 0.0214 Grad: 67280.5625 LR: 1.1661e-06 | Elapse: 1.50s
Epoch 9 [500/559] | Train Loss: 0.0320 Grad: 2366.7151 LR: 3.6575e-07 | Elapse: 627.09s
Epoch 9 [558/559] | Train Loss: 0.0303 Grad: 33299.4180 LR: 3.0086e-07 | Elapse: 700.43s
Epoch 9 [0/140] | Valid Loss: 0.3247 | Elapse: 1.65s
Epoch 9 [139/140] | Valid Loss: 0.1822 | Elapse: 171.60s
Epoch 9 - Train Loss: 0.0303 - Valid Loss: 0.6887 - Elapsed Time: 891.22s
- Epoch 9: Best model found with loss = 0.6887.
Epoch 10 [0/559] | Train Loss: 0.0396 Grad: 140012.0156 LR: 2.9979e-07 | Elapse: 1.47s
Epoch 10 [500/559] | Train Loss: 0.0399 Grad: 2683.7830 LR: 3.5668e-09 | Elapse: 627.50s
Epoch 10 [558/559] | Train Loss: 0.0374 Grad: 31579.2891 LR: 4.0097e-10 | Elapse: 699.21s
Epoch 10 [0/140] | Valid Loss: 0.3429 | Elapse: 1.75s
Epoch 10 [139/140] | Valid Loss: 0.1868 | Elapse: 170.59s
Epoch 10 - Train Loss: 0.0374 - Valid Loss: 0.6865 - Elapsed Time: 888.77s
Fold 1 | Time: 154.91min | Overall Evaluation Loss: 0.5993
Epoch 1 [0/559] | Train Loss: 0.4080 Grad: 124709.3203 LR: 4.0008e-07 | Elapse: 1.14s
Epoch 1 [500/559] | Train Loss: 0.1747 Grad: 1045.7129 LR: 9.7549e-06 | Elapse: 628.84s
Epoch 1 [558/559] | Train Loss: 0.1648 Grad: 1430.1704 LR: 1.0000e-05 | Elapse: 702.27s
Epoch 1 [0/140] | Valid Loss: 0.0024 | Elapse: 1.05s
Epoch 1 [139/140] | Valid Loss: 0.1646 | Elapse: 172.09s
Epoch 1 - Train Loss: 0.1648 - Valid Loss: 0.5391 - Elapsed Time: 892.89s
- Epoch 1: Best model found with loss = 0.5391.
Epoch 2 [0/559] | Train Loss: 0.3756 Grad: 93382.1719 LR: 1.0000e-05 | Elapse: 1.35s
Epoch 2 [500/559] | Train Loss: 0.1645 Grad: 1447.0669 LR: 9.7564e-06 | Elapse: 626.54s
Epoch 2 [558/559] | Train Loss: 0.1548 Grad: 2336.7964 LR: 9.6974e-06 | Elapse: 695.28s
Epoch 2 [0/140] | Valid Loss: 0.0028 | Elapse: 1.35s
Epoch 2 [139/140] | Valid Loss: 0.1744 | Elapse: 168.90s
Epoch 2 - Train Loss: 0.1548 - Valid Loss: 0.5480 - Elapsed Time: 882.83s
- Epoch 2: Best model found with loss = 0.5480.
Epoch 3 [0/559] | Train Loss: 0.3395 Grad: 155200.7188 LR: 9.6963e-06 | Elapse: 1.21s
Epoch 3 [500/559] | Train Loss: 0.1350 Grad: 1883.7952 LR: 8.9422e-06 | Elapse: 616.61s
Epoch 3 [558/559] | Train Loss: 0.1265 Grad: 3005.8718 LR: 8.8283e-06 | Elapse: 684.27s
Epoch 3 [0/140] | Valid Loss: 0.0033 | Elapse: 1.33s
Epoch 3 [139/140] | Valid Loss: 0.1848 | Elapse: 169.68s
Epoch 3 - Train Loss: 0.1265 - Valid Loss: 0.5793 - Elapsed Time: 872.76s
- Epoch 3: Best model found with loss = 0.5793.
Epoch 4 [0/559] | Train Loss: 0.2507 Grad: 184021.4375 LR: 8.8263e-06 | Elapse: 1.55s
Epoch 4 [500/559] | Train Loss: 0.0979 Grad: 2342.1018 LR: 7.6526e-06 | Elapse: 620.24s
Epoch 4 [558/559] | Train Loss: 0.0919 Grad: 3157.7532 LR: 7.4974e-06 | Elapse: 694.89s
Epoch 4 [0/140] | Valid Loss: 0.0033 | Elapse: 1.65s
Epoch 4 [139/140] | Valid Loss: 0.1921 | Elapse: 171.19s
Epoch 4 - Train Loss: 0.0919 - Valid Loss: 0.5966 - Elapsed Time: 884.77s
- Epoch 4: Best model found with loss = 0.5966.
Epoch 5 [0/559] | Train Loss: 0.1526 Grad: 191586.5938 LR: 7.4947e-06 | Elapse: 1.38s
Epoch 5 [500/559] | Train Loss: 0.0690 Grad: 2454.4775 LR: 6.0431e-06 | Elapse: 619.48s
Epoch 5 [558/559] | Train Loss: 0.0652 Grad: 3468.5071 LR: 5.8653e-06 | Elapse: 691.21s
Epoch 5 [0/140] | Valid Loss: 0.0035 | Elapse: 1.45s
Epoch 5 [139/140] | Valid Loss: 0.1998 | Elapse: 171.49s
Epoch 5 - Train Loss: 0.0652 - Valid Loss: 0.6213 - Elapsed Time: 881.09s
- Epoch 5: Best model found with loss = 0.6213.
Epoch 6 [0/559] | Train Loss: 0.0984 Grad: 176191.0312 LR: 5.8623e-06 | Elapse: 1.48s
Epoch 6 [500/559] | Train Loss: 0.0498 Grad: 2697.2048 LR: 4.3078e-06 | Elapse: 626.78s
Epoch 6 [558/559] | Train Loss: 0.0471 Grad: 3713.1016 LR: 4.1289e-06 | Elapse: 698.05s
Epoch 6 [0/140] | Valid Loss: 0.0033 | Elapse: 1.41s
Epoch 6 [139/140] | Valid Loss: 0.2037 | Elapse: 168.17s
Epoch 6 - Train Loss: 0.0471 - Valid Loss: 0.6357 - Elapsed Time: 885.22s
- Epoch 6: Best model found with loss = 0.6357.
Epoch 7 [0/559] | Train Loss: 0.0513 Grad: 126208.9375 LR: 4.1258e-06 | Elapse: 1.46s
Epoch 7 [500/559] | Train Loss: 0.0387 Grad: 2829.2466 LR: 2.6560e-06 | Elapse: 634.97s
Epoch 7 [558/559] | Train Loss: 0.0369 Grad: 3826.6626 LR: 2.4976e-06 | Elapse: 705.57s
Epoch 7 [0/140] | Valid Loss: 0.0033 | Elapse: 0.98s
Epoch 7 [139/140] | Valid Loss: 0.2071 | Elapse: 172.22s
Epoch 7 - Train Loss: 0.0369 - Valid Loss: 0.6424 - Elapsed Time: 896.53s
- Epoch 7: Best model found with loss = 0.6424.
Epoch 8 [0/559] | Train Loss: 0.0380 Grad: 107768.7891 LR: 2.4949e-06 | Elapse: 1.63s
Epoch 8 [500/559] | Train Loss: 0.0336 Grad: 2959.4180 LR: 1.2869e-06 | Elapse: 626.93s
Epoch 8 [558/559] | Train Loss: 0.0322 Grad: 3683.2998 LR: 1.1681e-06 | Elapse: 702.47s
Epoch 8 [0/140] | Valid Loss: 0.0034 | Elapse: 1.14s
Epoch 8 [139/140] | Valid Loss: 0.2092 | Elapse: 171.83s
Epoch 8 - Train Loss: 0.0322 - Valid Loss: 0.6436 - Elapsed Time: 892.56s
- Epoch 8: Best model found with loss = 0.6436.
Epoch 9 [0/559] | Train Loss: 0.0356 Grad: 110887.7266 LR: 1.1661e-06 | Elapse: 1.26s
Epoch 9 [500/559] | Train Loss: 0.0349 Grad: 2969.2019 LR: 3.6575e-07 | Elapse: 618.59s
Epoch 9 [558/559] | Train Loss: 0.0333 Grad: 3657.1890 LR: 3.0086e-07 | Elapse: 689.52s
Epoch 9 [0/140] | Valid Loss: 0.0034 | Elapse: 0.85s
Epoch 9 [139/140] | Valid Loss: 0.2080 | Elapse: 169.88s
Epoch 9 - Train Loss: 0.0333 - Valid Loss: 0.6454 - Elapsed Time: 877.97s
- Epoch 9: Best model found with loss = 0.6454.
Epoch 10 [0/559] | Train Loss: 0.0413 Grad: 124596.9844 LR: 2.9979e-07 | Elapse: 1.29s
Epoch 10 [500/559] | Train Loss: 0.0474 Grad: 3126.1436 LR: 3.5668e-09 | Elapse: 627.28s
Epoch 10 [558/559] | Train Loss: 0.0448 Grad: 4568.4751 LR: 4.0097e-10 | Elapse: 698.02s
Epoch 10 [0/140] | Valid Loss: 0.0033 | Elapse: 1.65s
Epoch 10 [139/140] | Valid Loss: 0.2082 | Elapse: 171.99s
Epoch 10 - Train Loss: 0.0448 - Valid Loss: 0.6580 - Elapsed Time: 888.30s
- Epoch 10: Best model found with loss = 0.6580.
Fold 2 | Time: 148.58min | Overall Evaluation Loss: 0.5356
Epoch 1 [0/559] | Train Loss: 0.3735 Grad: 136774.1406 LR: 4.0008e-07 | Elapse: 1.12s
Epoch 1 [500/559] | Train Loss: 0.1727 Grad: 19389.6543 LR: 9.7549e-06 | Elapse: 623.82s
Epoch 1 [558/559] | Train Loss: 0.1621 Grad: 33160.3281 LR: 1.0000e-05 | Elapse: 697.46s
Epoch 1 [0/140] | Valid Loss: 0.0017 | Elapse: 1.14s
Epoch 1 [139/140] | Valid Loss: 0.1746 | Elapse: 169.70s
Epoch 1 - Train Loss: 0.1621 - Valid Loss: 0.5274 - Elapsed Time: 887.75s
- Epoch 1: Best model found with loss = 0.5274.
Epoch 2 [0/559] | Train Loss: 0.3857 Grad: 82156.1875 LR: 1.0000e-05 | Elapse: 1.27s
Epoch 2 [500/559] | Train Loss: 0.1630 Grad: 29308.9199 LR: 9.7564e-06 | Elapse: 623.37s
Epoch 2 [558/559] | Train Loss: 0.1524 Grad: 44503.8945 LR: 9.6974e-06 | Elapse: 693.21s
Epoch 2 [0/140] | Valid Loss: 0.0018 | Elapse: 1.15s
Epoch 2 [139/140] | Valid Loss: 0.1843 | Elapse: 176.11s
Epoch 2 - Train Loss: 0.1524 - Valid Loss: 0.5781 - Elapsed Time: 889.88s
- Epoch 2: Best model found with loss = 0.5781.
Epoch 3 [0/559] | Train Loss: 0.3332 Grad: 135450.9531 LR: 9.6963e-06 | Elapse: 1.49s
Epoch 3 [500/559] | Train Loss: 0.1318 Grad: 32993.6094 LR: 8.9422e-06 | Elapse: 622.89s
Epoch 3 [558/559] | Train Loss: 0.1228 Grad: 51153.7461 LR: 8.8283e-06 | Elapse: 691.03s
Epoch 3 [0/140] | Valid Loss: 0.0020 | Elapse: 1.04s
Epoch 3 [139/140] | Valid Loss: 0.1926 | Elapse: 168.78s
Epoch 3 - Train Loss: 0.1228 - Valid Loss: 0.6165 - Elapsed Time: 880.74s
- Epoch 3: Best model found with loss = 0.6165.
Epoch 4 [0/559] | Train Loss: 0.2050 Grad: 158852.4688 LR: 8.8263e-06 | Elapse: 1.24s
Epoch 4 [500/559] | Train Loss: 0.0946 Grad: 32502.8730 LR: 7.6526e-06 | Elapse: 611.44s
Epoch 4 [558/559] | Train Loss: 0.0882 Grad: 52789.3359 LR: 7.4974e-06 | Elapse: 684.08s
Epoch 4 [0/140] | Valid Loss: 0.0021 | Elapse: 1.26s
Epoch 4 [139/140] | Valid Loss: 0.2005 | Elapse: 173.50s
Epoch 4 - Train Loss: 0.0882 - Valid Loss: 0.6403 - Elapsed Time: 878.81s
- Epoch 4: Best model found with loss = 0.6403.
Epoch 5 [0/559] | Train Loss: 0.1045 Grad: 160419.8594 LR: 7.4947e-06 | Elapse: 1.13s
Epoch 5 [500/559] | Train Loss: 0.0674 Grad: 33515.8281 LR: 6.0431e-06 | Elapse: 622.62s
Epoch 5 [558/559] | Train Loss: 0.0630 Grad: 48679.0625 LR: 5.8653e-06 | Elapse: 694.96s
Epoch 5 [0/140] | Valid Loss: 0.0022 | Elapse: 1.26s
Epoch 5 [139/140] | Valid Loss: 0.2054 | Elapse: 174.00s
Epoch 5 - Train Loss: 0.0630 - Valid Loss: 0.6581 - Elapsed Time: 889.52s
- Epoch 5: Best model found with loss = 0.6581.
Epoch 6 [0/559] | Train Loss: 0.0513 Grad: 123881.2109 LR: 5.8623e-06 | Elapse: 1.20s
Epoch 6 [500/559] | Train Loss: 0.0489 Grad: 34166.4883 LR: 4.3078e-06 | Elapse: 619.33s
Epoch 6 [558/559] | Train Loss: 0.0459 Grad: 46318.1602 LR: 4.1289e-06 | Elapse: 692.04s
Epoch 6 [0/140] | Valid Loss: 0.0022 | Elapse: 1.06s
Epoch 6 [139/140] | Valid Loss: 0.2085 | Elapse: 175.60s
Epoch 6 - Train Loss: 0.0459 - Valid Loss: 0.6727 - Elapsed Time: 888.27s
- Epoch 6: Best model found with loss = 0.6727.
Epoch 7 [0/559] | Train Loss: 0.0245 Grad: 69471.7734 LR: 4.1258e-06 | Elapse: 1.23s
Epoch 7 [500/559] | Train Loss: 0.0379 Grad: 33260.8320 LR: 2.6560e-06 | Elapse: 633.33s
Epoch 7 [558/559] | Train Loss: 0.0358 Grad: 43805.9805 LR: 2.4976e-06 | Elapse: 707.50s
Epoch 7 [0/140] | Valid Loss: 0.0023 | Elapse: 1.22s
Epoch 7 [139/140] | Valid Loss: 0.2125 | Elapse: 173.57s
Epoch 7 - Train Loss: 0.0358 - Valid Loss: 0.6797 - Elapsed Time: 901.75s
- Epoch 7: Best model found with loss = 0.6797.
Epoch 8 [0/559] | Train Loss: 0.0170 Grad: 45662.7891 LR: 2.4949e-06 | Elapse: 1.28s
Epoch 8 [500/559] | Train Loss: 0.0332 Grad: 33284.9766 LR: 1.2869e-06 | Elapse: 636.77s
Epoch 8 [558/559] | Train Loss: 0.0315 Grad: 45330.4883 LR: 1.1681e-06 | Elapse: 709.71s
Epoch 8 [0/140] | Valid Loss: 0.0023 | Elapse: 1.45s
Epoch 8 [139/140] | Valid Loss: 0.2158 | Elapse: 172.70s
Epoch 8 - Train Loss: 0.0315 - Valid Loss: 0.6806 - Elapsed Time: 903.01s
- Epoch 8: Best model found with loss = 0.6806.
Epoch 9 [0/559] | Train Loss: 0.0181 Grad: 55811.3711 LR: 1.1661e-06 | Elapse: 1.26s
Epoch 9 [500/559] | Train Loss: 0.0337 Grad: 36090.6758 LR: 3.6575e-07 | Elapse: 622.66s
Epoch 9 [558/559] | Train Loss: 0.0319 Grad: 40806.4766 LR: 3.0086e-07 | Elapse: 695.00s
Epoch 9 [0/140] | Valid Loss: 0.0024 | Elapse: 1.55s
Epoch 9 [139/140] | Valid Loss: 0.2160 | Elapse: 172.99s
Epoch 9 - Train Loss: 0.0319 - Valid Loss: 0.6900 - Elapsed Time: 888.59s
- Epoch 9: Best model found with loss = 0.6900.
Epoch 10 [0/559] | Train Loss: 0.0291 Grad: 108929.7500 LR: 2.9979e-07 | Elapse: 1.67s
Epoch 10 [500/559] | Train Loss: 0.0408 Grad: 33068.8359 LR: 3.5668e-09 | Elapse: 628.66s
Epoch 10 [558/559] | Train Loss: 0.0381 Grad: 40680.0781 LR: 4.0097e-10 | Elapse: 701.00s
Epoch 10 [0/140] | Valid Loss: 0.0026 | Elapse: 1.65s
Epoch 10 [139/140] | Valid Loss: 0.2175 | Elapse: 172.10s
Epoch 10 - Train Loss: 0.0381 - Valid Loss: 0.6948 - Elapsed Time: 893.76s
- Epoch 10: Best model found with loss = 0.6948.
Fold 3 | Time: 149.63min | Overall Evaluation Loss: 0.4956
Epoch 1 [0/559] | Train Loss: 0.0050 Grad: 2809.6936 LR: 4.0008e-07 | Elapse: 1.47s
Epoch 1 [500/559] | Train Loss: 0.1740 Grad: 374.4365 LR: 9.7549e-06 | Elapse: 619.57s
Epoch 1 [558/559] | Train Loss: 0.1637 Grad: 36396.9766 LR: 1.0000e-05 | Elapse: 689.00s
Epoch 1 [0/140] | Valid Loss: 0.4124 | Elapse: 1.45s
Epoch 1 [139/140] | Valid Loss: 0.1685 | Elapse: 171.89s
Epoch 1 - Train Loss: 0.1637 - Valid Loss: 0.5389 - Elapsed Time: 881.37s
- Epoch 1: Best model found with loss = 0.5389.
Epoch 2 [0/559] | Train Loss: 0.0050 Grad: 1995.7759 LR: 1.0000e-05 | Elapse: 1.59s
Epoch 2 [500/559] | Train Loss: 0.1633 Grad: 583.9670 LR: 9.7564e-06 | Elapse: 624.89s
Epoch 2 [558/559] | Train Loss: 0.1530 Grad: 46425.1641 LR: 9.6974e-06 | Elapse: 694.86s
Epoch 2 [0/140] | Valid Loss: 0.4686 | Elapse: 1.01s
Epoch 2 [139/140] | Valid Loss: 0.1789 | Elapse: 167.87s
Epoch 2 - Train Loss: 0.1530 - Valid Loss: 0.5844 - Elapsed Time: 882.92s
- Epoch 2: Best model found with loss = 0.5844.
Epoch 3 [0/559] | Train Loss: 0.0053 Grad: 3130.1858 LR: 9.6963e-06 | Elapse: 1.07s
Epoch 3 [500/559] | Train Loss: 0.1322 Grad: 783.8658 LR: 8.9422e-06 | Elapse: 627.07s
Epoch 3 [558/559] | Train Loss: 0.1232 Grad: 45816.0273 LR: 8.8283e-06 | Elapse: 699.61s
Epoch 3 [0/140] | Valid Loss: 0.4931 | Elapse: 1.25s
Epoch 3 [139/140] | Valid Loss: 0.1861 | Elapse: 167.99s
Epoch 3 - Train Loss: 0.1232 - Valid Loss: 0.6180 - Elapsed Time: 887.79s
- Epoch 3: Best model found with loss = 0.6180.
Epoch 4 [0/559] | Train Loss: 0.0056 Grad: 4049.7507 LR: 8.8263e-06 | Elapse: 1.48s
Epoch 4 [500/559] | Train Loss: 0.0952 Grad: 915.9907 LR: 7.6526e-06 | Elapse: 621.37s
Epoch 4 [558/559] | Train Loss: 0.0887 Grad: 42097.1250 LR: 7.4974e-06 | Elapse: 692.63s
Epoch 4 [0/140] | Valid Loss: 0.4977 | Elapse: 1.44s
Epoch 4 [139/140] | Valid Loss: 0.1917 | Elapse: 166.80s
Epoch 4 - Train Loss: 0.0887 - Valid Loss: 0.6386 - Elapsed Time: 879.67s
- Epoch 4: Best model found with loss = 0.6386.
Epoch 5 [0/559] | Train Loss: 0.0056 Grad: 4627.5327 LR: 7.4947e-06 | Elapse: 1.31s
Epoch 5 [500/559] | Train Loss: 0.0673 Grad: 1042.5446 LR: 6.0431e-06 | Elapse: 623.91s
Epoch 5 [558/559] | Train Loss: 0.0628 Grad: 39756.8047 LR: 5.8653e-06 | Elapse: 695.74s
Epoch 5 [0/140] | Valid Loss: 0.4978 | Elapse: 1.65s
Epoch 5 [139/140] | Valid Loss: 0.1959 | Elapse: 172.59s
Epoch 5 - Train Loss: 0.0628 - Valid Loss: 0.6606 - Elapsed Time: 888.42s
- Epoch 5: Best model found with loss = 0.6606.
Epoch 6 [0/559] | Train Loss: 0.0055 Grad: 4887.3267 LR: 5.8623e-06 | Elapse: 1.38s
Epoch 6 [500/559] | Train Loss: 0.0492 Grad: 1069.9318 LR: 4.3078e-06 | Elapse: 619.50s
Epoch 6 [558/559] | Train Loss: 0.0460 Grad: 38461.5625 LR: 4.1289e-06 | Elapse: 692.72s
Epoch 6 [0/140] | Valid Loss: 0.5020 | Elapse: 1.05s
Epoch 6 [139/140] | Valid Loss: 0.1990 | Elapse: 174.79s
Epoch 6 - Train Loss: 0.0460 - Valid Loss: 0.6746 - Elapsed Time: 887.61s
- Epoch 6: Best model found with loss = 0.6746.
Epoch 7 [0/559] | Train Loss: 0.0054 Grad: 5169.7212 LR: 4.1258e-06 | Elapse: 1.07s
Epoch 7 [500/559] | Train Loss: 0.0381 Grad: 1063.5841 LR: 2.6560e-06 | Elapse: 621.07s
Epoch 7 [558/559] | Train Loss: 0.0359 Grad: 35426.7031 LR: 2.4976e-06 | Elapse: 693.61s
Epoch 7 [0/140] | Valid Loss: 0.5056 | Elapse: 1.28s
Epoch 7 [139/140] | Valid Loss: 0.2010 | Elapse: 169.21s
Epoch 7 - Train Loss: 0.0359 - Valid Loss: 0.6811 - Elapsed Time: 883.41s
- Epoch 7: Best model found with loss = 0.6811.
Epoch 8 [0/559] | Train Loss: 0.0054 Grad: 5201.8013 LR: 2.4949e-06 | Elapse: 1.16s
Epoch 8 [500/559] | Train Loss: 0.0335 Grad: 1033.7025 LR: 1.2869e-06 | Elapse: 621.26s
Epoch 8 [558/559] | Train Loss: 0.0316 Grad: 32125.7207 LR: 1.1681e-06 | Elapse: 691.80s
Epoch 8 [0/140] | Valid Loss: 0.5071 | Elapse: 1.45s
Epoch 8 [139/140] | Valid Loss: 0.2006 | Elapse: 174.00s
Epoch 8 - Train Loss: 0.0316 - Valid Loss: 0.6861 - Elapsed Time: 885.99s
- Epoch 8: Best model found with loss = 0.6861.
Epoch 9 [0/559] | Train Loss: 0.0054 Grad: 5315.4302 LR: 1.1661e-06 | Elapse: 1.47s
Epoch 9 [500/559] | Train Loss: 0.0337 Grad: 1095.0151 LR: 3.6575e-07 | Elapse: 622.67s
Epoch 9 [558/559] | Train Loss: 0.0319 Grad: 27265.7305 LR: 3.0086e-07 | Elapse: 694.01s
Epoch 9 [0/140] | Valid Loss: 0.4932 | Elapse: 1.35s
Epoch 9 [139/140] | Valid Loss: 0.1994 | Elapse: 174.70s
Epoch 9 - Train Loss: 0.0319 - Valid Loss: 0.6887 - Elapsed Time: 888.81s
- Epoch 9: Best model found with loss = 0.6887.
Epoch 10 [0/559] | Train Loss: 0.0052 Grad: 5499.5928 LR: 2.9979e-07 | Elapse: 1.36s
Epoch 10 [500/559] | Train Loss: 0.0392 Grad: 1228.2296 LR: 3.5668e-09 | Elapse: 626.25s
Epoch 10 [558/559] | Train Loss: 0.0367 Grad: 28973.5898 LR: 4.0097e-10 | Elapse: 696.89s
Epoch 10 [0/140] | Valid Loss: 0.5141 | Elapse: 1.16s
Epoch 10 [139/140] | Valid Loss: 0.2049 | Elapse: 174.49s
Epoch 10 - Train Loss: 0.0367 - Valid Loss: 0.6837 - Elapsed Time: 891.96s
Fold 4 | Time: 149.09min | Overall Evaluation Loss: 0.4522