waveletdeboshir's picture
Add git link
0111641 verified
metadata
license: apache-2.0
language:
  - ru
library_name: transformers
pipeline_tag: automatic-speech-recognition
tags:
  - asr
  - Pytorch
  - pruned
  - audio
  - automatic-speech-recognition
metrics:
  - cer
  - wer
model-index:
  - name: Whisper Small Pruned for Russian
    results:
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 15.0 (Russian part, test)
          type: mozilla-foundation/common_voice_15_0
          args: ru
        metrics:
          - name: WER
            type: wer
            value: 24.98
      - task:
          name: Speech Recognition
          type: automatic-speech-recognition
        dataset:
          name: Common Voice 15.0 (Russian part, test)
          type: mozilla-foundation/common_voice_15_0
          args: ru
        metrics:
          - name: WER (without punctuation)
            type: wer
            value: 17.48

Whisper-small-ru-pruned

Model info

This is a pruned version of openai/whisper-small model with only russian tokens left. Pruning was made without any fine-tuning. Method from this post was used.

Size

Only 10% tokens was left including special whisper tokens (no language tokens except <|ru|> and <|en|>, no timestamp tokens), 200 most popular tokens from tokenizer and 4000 most popular Russian tokens computed by tokenization of russian text corpus.

Model size is 15% less then original whisper-small:

openai/whisper-small waveletdeboshir/whisper-small-ru-pruned
n of parameters 242 M 205 M
n of parameters (with proj_out layer) 281 M 208 M
model file size 967 Mb 834 Mb
vocab_size 51865 4207

Usage

Model can be used as an original whisper:

>>> from transformers import WhisperProcessor, WhisperForConditionalGeneration
>>> import torchaudio

>>> # load audio
>>> wav, sr = torchaudio.load("audio.wav")

>>> # load model and processor
>>> processor = WhisperProcessor.from_pretrained("waveletdeboshir/whisper-small-ru-pruned")
>>> model = WhisperForConditionalGeneration.from_pretrained("waveletdeboshir/whisper-small-ru-pruned")

>>> input_features = processor(wav[0], sampling_rate=sr, return_tensors="pt").input_features 

>>> # generate token ids
>>> predicted_ids = model.generate(input_features)
>>> # decode token ids to text
>>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=False)
['<|startoftranscript|><|ru|><|transcribe|><|notimestamps|> Начинаем работу.<|endoftext|>']

The context tokens can be removed from the start of the transcription by setting skip_special_tokens=True.

Other pruned whisper models

Metrics

metric dataset openai/whisper-small waveletdeboshir/whisper-small-ru-pruned
WER* golos-test-crowd 0.3358 0.3471
CER* golos-test-crowd 0.1561 0.1444
WER* common_voice_15_0_test 0.1749 0.1748
WER common_voice_15_0_test 0.2492 0.2498
*Metrics were computed after text normalization

You can fine-tune this model on your data to achive better performance.

Colab for vocab pruning

Check https://github.com/waveletdeboshir/whisper-lang-remover