openai

/

whisper-small

like 295

Follow

OpenAI 2,448

Remove single quotes ' (id 6) and hyphens - (id 12) from suppress_tokens. These tokens should not be suppressed during generation. They are accepted as valid generated tokens in the official Whisper repo:
https://github.com/openai/whisper/blob/eff383b27b783e280c089475852ba83f20f64998/whisper/tokenizer.py#L258

Check that we're removing the right tokens:

from transformers import WhisperTokenizer

tokenizer = WhisperTokenizer.from_pretrained("openai/whisper-small")

print(tokenizer.decode(6))
print(tokenizer.decode(12))

Print Output:
```
'