update README + add evaluation

README.md

@@ -9,6 +9,7 @@ datasets:
 - mozilla-foundation/common_voice_11_0
 metrics:
 - wer
+- cer
 model-index:
 - name: Whisper Large Chinese (Mandarin)
   results:
@@ -19,76 +20,81 @@ model-index:
       name: mozilla-foundation/common_voice_11_0 zh-CN
       type: mozilla-foundation/common_voice_11_0
       config: zh-CN
-      split: validation[:1000]
+      split: test
       args: zh-CN
     metrics:
-    - name: Wer
+    - name: WER
       type: wer
-      value: 51.67420814479639
+      value: 55.02141421204441
+    - name: CER
+      type: cer
+      value: 9.550758567294045
+  - task:
+      name: Automatic Speech Recognition
+      type: automatic-speech-recognition
+    dataset:
+      name: google/fleurs cmn_hans_cn
+      type: google/fleurs
+      config: cmn_hans_cn
+      split: test
+      args: cmn_hans_cn
+    metrics:
+    - name: WER
+      type: wer
+      value: 70.62596203181118
+    - name: CER
+      type: cer
+      value: 11.761282471826888
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
 # Whisper Large Chinese (Mandarin)
 
-This model is a fine-tuned version of [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) on the mozilla-foundation/common_voice_11_0 zh-CN dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.2435
-- Wer: 51.6742
-- Cer: 8.5279
-
-## Model description
+This model is a fine-tuned version of [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) on Chinese (Mandarin) using the train and validation splits of [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0). Not all validation split data were used during training, I extracted 1k samples from the validation split to be used for evaluation during fine-tuning. When using this model, make sure that your speech input is sampled at 16kHz.
 
-More information needed
+## Usage
 
-## Intended uses & limitations
+```python
 
-More information needed
+from transformers import pipeline
 
-## Training and evaluation data
+transcriber = pipeline(
+  "automatic-speech-recognition", 
+  model="jonatasgrosman/whisper-large-zh-cv11"
+)
 
-More information needed
+transcriber.model.config.forced_decoder_ids = (
+  transcriber.tokenizer.get_decoder_prompt_ids(
+    language="zh" 
+    task="transcribe"
+  )
+)
 
-## Training procedure
+transcription = transcriber("path/to/my_audio.wav")
 
-### Training hyperparameters
+```
 
-The following hyperparameters were used during training:
-- learning_rate: 5e-06
-- train_batch_size: 16
-- eval_batch_size: 8
-- seed: 42
-- gradient_accumulation_steps: 2
-- total_train_batch_size: 32
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_steps: 2000
-- training_steps: 20000
-- mixed_precision_training: Native AMP
+## Evaluation
 
-### Training results
+We perform evaluation of the model using the test split of two datasets, the [Common Voice 11](https://huggingface.co/datasets/mozilla-foundation/common_voice_11_0) (same dataset used for the fine-tuning) and the [Fleurs](https://huggingface.co/datasets/google/fleurs) (dataset not seen during the fine-tuning). As Whisper can transcribe casing and punctuation, I performed the model evaluation in 2 different scenarios, one using the raw text and the other using the normalized text (lowercase + removal of punctuations). Additionally, for the Fleurs dataset, I evaluated the model in a scenario where there are no transcriptions of numerical values since the way these values are described in this dataset is different from how they are described in the dataset used in fine-tuning (Common Voice), so it is expected that this difference in the way of describing numerical values will affect the performance of the model for this type of transcription in Fleurs.
 
-| Training Loss | Epoch | Step  | Validation Loss | Wer     | Cer     |
-|:-------------:|:-----:|:-----:|:---------------:|:-------:|:-------:|
-| 0.3314        | 0.83  | 1000  | 0.2110          | 65.7014 | 10.8047 |
-| 0.2747        | 1.66  | 2000  | 0.2005          | 58.1900 | 9.4191  |
-| 0.1989        | 2.49  | 3000  | 0.1983          | 56.1991 | 9.0939  |
-| 0.1142        | 3.31  | 4000  | 0.2076          | 55.0226 | 9.1589  |
-| 0.0747        | 4.14  | 5000  | 0.2131          | 56.3801 | 9.0483  |
-| 0.0709        | 4.97  | 6000  | 0.2165          | 54.6606 | 8.9768  |
-| 0.0432        | 5.8   | 7000  | 0.2222          | 54.0271 | 8.9508  |
-| 0.0261        | 6.63  | 8000  | 0.2299          | 54.4796 | 9.0353  |
-| 0.0152        | 7.46  | 9000  | 0.2290          | 52.7602 | 8.8076  |
-| 0.0054        | 8.28  | 10000 | 0.2435          | 51.6742 | 8.5279  |
-| 0.0028        | 9.11  | 11000 | 0.2421          | 53.0317 | 8.9833  |
-| 0.0045        | 9.94  | 12000 | 0.2462          | 52.9412 | 8.7751  |
-| 0.0016        | 10.77 | 13000 | 0.2501          | 52.3077 | 8.9573  |
+### Common Voice 11
 
+| | CER | WER |
+| --- | --- | --- |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) | 9.31 | 55.94 |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) + text normalization | 9.55 | 55.02 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) | 33.33 | 101.80 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) + text normalization | 29.90 | 95.91 |
 
-### Framework versions
+### Fleurs
 
-- Transformers 4.26.0.dev0
-- Pytorch 1.13.1+cu117
-- Datasets 2.7.1.dev0
-- Tokenizers 0.13.2
+| | CER | WER |
+| --- | --- | --- |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) | 15.00 | 93.45 |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) + text normalization | 11.76 | 70.63 |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) + keep only non-numeric samples | 10.95 | 87.91 |
+| [jonatasgrosman/whisper-large-zh-cv11](https://huggingface.co/jonatasgrosman/whisper-large-zh-cv11) + text normalization + keep only non-numeric samples | 7.83 | 62.12 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) | 23.49 | 101.28 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) + text normalization | 17.58 | 83.22 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) + keep only non-numeric samples | 21.03 | 101.95 |
+| [openai/whisper-large-v2](https://huggingface.co/openai/whisper-large-v2) + text normalization + keep only non-numeric samples | 15.22 | 79.28 |

evaluation_cv11_test.json

@@ -2,6 +2,8 @@
     "raw": {
         "cer": 0.09311360578930278,
         "wer": 0.5594405594405595,
+        "non_numeric_samples_cer": 0.09311360578930278,
+        "non_numeric_samples_wer": 0.5594405594405595,
         "references": [
             "否",
             "宋朝末年年间定居粉岭围。",
@@ -21172,6 +21174,8 @@
     "normalized": {
         "cer": 0.09550758567294045,
         "wer": 0.5502141421204441,
+        "non_numeric_samples_cer": 0.09550758567294045,
+        "non_numeric_samples_wer": 0.5502141421204441,
         "references": [
             "否",
             "宋朝末年年间定居粉岭围",

evaluation_fleurs_test.json

@@ -2,6 +2,8 @@
     "raw": {
         "cer": 0.1500187149095446,
         "wer": 0.9344808439755691,
+        "non_numeric_samples_cer": 0.10947220549869556,
+        "non_numeric_samples_wer": 0.8790983606557377,
         "references": [
             "1940 年 8 月 15 日，盟军攻入法国南部，这次进攻被称为“龙骑兵行动”。",
             "该群岛位于南极半岛以北 120 公里处。最大的岛屿是乔治国王岛，这里是“繁星村（Villa Las Estrellas）”的定居点。",
@@ -1900,6 +1902,8 @@
     "normalized": {
         "cer": 0.11761282471826888,
         "wer": 0.7062596203181118,
+        "non_numeric_samples_cer": 0.07828692280578076,
+        "non_numeric_samples_wer": 0.6211941478845393,
         "references": [
             "1940 年 8 月 15 日 盟军攻入法国南部 这次进攻被称为 龙骑兵行动",
             "该群岛位于南极半岛以北 120 公里处 最大的岛屿是乔治国王岛 这里是 繁星村 villa las estrellas 的定居点",

evaluation_whisper-large-v2_cv11_test.json

@@ -2,6 +2,8 @@
     "raw": {
         "cer": 0.33334879621468666,
         "wer": 1.0180495180495182,
+        "non_numeric_samples_cer": 0.33334879621468666,
+        "non_numeric_samples_wer": 1.0180495180495182,
         "references": [
             "否",
             "宋朝末年年间定居粉岭围。",
@@ -21172,6 +21174,8 @@
     "normalized": {
         "cer": 0.299039578225524,
         "wer": 0.9590944847478368,
+        "non_numeric_samples_cer": 0.299039578225524,
+        "non_numeric_samples_wer": 0.9590944847478368,
         "references": [
             "否",
             "宋朝末年年间定居粉岭围",

evaluation_whisper-large-v2_fleurs_test.json

@@ -2,6 +2,8 @@
     "raw": {
         "cer": 0.23488459139114162,
         "wer": 1.0127706829539145,
+        "non_numeric_samples_cer": 0.21031507124222357,
+        "non_numeric_samples_wer": 1.0194672131147542,
         "references": [
             "1940 年 8 月 15 日，盟军攻入法国南部，这次进攻被称为“龙骑兵行动”。",
             "该群岛位于南极半岛以北 120 公里处。最大的岛屿是乔治国王岛，这里是“繁星村（Villa Las Estrellas）”的定居点。",
@@ -1900,6 +1902,8 @@
     "normalized": {
         "cer": 0.17581080366118196,
         "wer": 0.8322216521292971,
+        "non_numeric_samples_cer": 0.15216778286922805,
+        "non_numeric_samples_wer": 0.7928034796362199,
         "references": [
             "1940 年 8 月 15 日 盟军攻入法国南部 这次进攻被称为 龙骑兵行动",
             "该群岛位于南极半岛以北 120 公里处 最大的岛屿是乔治国王岛 这里是 繁星村 villa las estrellas 的定居点",