distilhubert-finetuned-gtzan
This model is a fine-tuned version of ntu-spml/distilhubert on the GTZAN dataset. It achieves the following results on the evaluation set on best epoch:
- Loss: 0.7305
- Accuracy: 0.9
Model description
Distilhubert is distilled version of the HuBERT and pretrained on data set with 16k frequency.
Architecture of this model is CTC or Connectionist Temporal Classification is a technique that is used with encoder-only transformer.
Training and evaluation data
Training + Evaluation data set is GTZAN which is a popular dataset of 999 songs for music genre classification.
Each song is a 30-second clip from one of 10 genres of music, spanning disco to metal.
Train set is 899 songs and Evaluation set is 100 songs remainings.
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 4
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 35
- mixed_precision_training: Native AMP
Training results
Training Loss | Epoch | Step | Validation Loss | Accuracy |
---|---|---|---|---|
2.1728 | 1.0 | 225 | 2.0896 | 0.42 |
1.4211 | 2.0 | 450 | 1.4951 | 0.55 |
1.2155 | 3.0 | 675 | 1.0669 | 0.72 |
1.0175 | 4.0 | 900 | 0.8862 | 0.69 |
0.3516 | 5.0 | 1125 | 0.6265 | 0.83 |
0.6135 | 6.0 | 1350 | 0.6485 | 0.78 |
0.0807 | 7.0 | 1575 | 0.6567 | 0.78 |
0.0303 | 8.0 | 1800 | 0.7615 | 0.83 |
0.2663 | 9.0 | 2025 | 0.6612 | 0.86 |
0.0026 | 10.0 | 2250 | 0.8354 | 0.85 |
0.0337 | 11.0 | 2475 | 0.6768 | 0.87 |
0.0013 | 12.0 | 2700 | 0.7718 | 0.87 |
0.001 | 13.0 | 2925 | 0.7570 | 0.88 |
0.0008 | 14.0 | 3150 | 0.8170 | 0.89 |
0.0006 | 15.0 | 3375 | 0.7920 | 0.89 |
0.0005 | 16.0 | 3600 | 0.9859 | 0.83 |
0.0004 | 17.0 | 3825 | 0.8190 | 0.9 |
0.0003 | 18.0 | 4050 | 0.7305 | 0.9 |
0.0003 | 19.0 | 4275 | 0.8025 | 0.88 |
0.0002 | 20.0 | 4500 | 0.8208 | 0.87 |
0.0003 | 21.0 | 4725 | 0.7358 | 0.88 |
0.0002 | 22.0 | 4950 | 0.8681 | 0.87 |
0.0002 | 23.0 | 5175 | 0.7831 | 0.9 |
0.0003 | 24.0 | 5400 | 0.8583 | 0.88 |
0.0002 | 25.0 | 5625 | 0.8138 | 0.88 |
0.0002 | 26.0 | 5850 | 0.7871 | 0.89 |
0.0002 | 27.0 | 6075 | 0.8893 | 0.88 |
0.0002 | 28.0 | 6300 | 0.8284 | 0.89 |
0.0001 | 29.0 | 6525 | 0.8388 | 0.89 |
0.0001 | 30.0 | 6750 | 0.8305 | 0.9 |
0.0001 | 31.0 | 6975 | 0.8377 | 0.88 |
0.0153 | 32.0 | 7200 | 0.8496 | 0.88 |
0.0001 | 33.0 | 7425 | 0.8381 | 0.88 |
0.0001 | 34.0 | 7650 | 0.8440 | 0.88 |
0.0001 | 35.0 | 7875 | 0.8458 | 0.88 |
Framework versions
- Transformers 4.29.2
- Pytorch 1.13.1+cu117
- Datasets 2.12.0
- Tokenizers 0.13.3
- Downloads last month
- 51
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social
visibility and check back later, or deploy to Inference Endpoints (dedicated)
instead.