Query regarding Model's output
I have query regarding a specific behaviour in model's output. Consider the two audio inputs to the model:
Input 1: "हेलो क्या मेरी बात रजत से हो रही है "
Input 2: "हेलो pause क्या मेरी बात रजत से हो रही है "
pause maybe of about 1-2 sec.
In second case I am getting model's output as: "हेलो"
The model is not transcribing anything after the pause.
This behaviour is also shown if I give a long audio. In such case, model is missing one or two word after the pause everywhere in the audio.
Can you tell me what could be reason behind such behaviour? I am unable to find out what could be the reason. A similar model (trained on the shrutilipi data) by ai4bharat shows no such behaviour. Is it because the architecture is different? What could be the possible issue in your opinion?
I am asking these questions because I was expecting whisper to show better results.
Hi @tttarun ,
Thanks for your query on long audios. When trying to transcribe a long audio, it would be beneficial to use an external voice activity detection module (Eg: silero-vad) and thereafter pass the segments to whisper for inference. This is because, though whisper's silence detection is quite good, it does miss some segments at times.
Regarding the model not transcribing after a long pause, I haven't come across such an issue with the test audios at my end. Request you to upload the audio sample here for which this issue exists. The issue can be looked into better with access to the audio sample.
Thanks
Hi
@vasista22
, thanks for the reply.
Here I am attaching the sample audio for reproduction of the query:
input 1:
@tttarun ,
Thanks for the speech samples.
At chunk_length_s=30 the transcription is "हेलो" like you've mentioned.
However, when I've reduced it to chunk_length_s=5, the output was: "हेलोबात हो रही है". Though not completely accurate the length of the transcription has improved with a shorter chunk_length_s.
There is no clear reason according to me for this behavior from the model.
As every model has a few examples it isn't robust to, this might be one such example for this model.
I would be keen to know if anyone comes up with a way to improve the transcription of this model over such an example by any means.
hi
@vasista22
I think I got something to improve model's transcription over the sample example.. The audios need to be pre-processed to remove the silences (or the pause) before sending them for transcription. Found this ffmpeg code for pre-processing:
ffmpeg -y -i input.wav -af "silenceremove=start_periods=0:stop_periods=-1:stop_duration=0.7:stop_threshold=-27dB" processed_input.wav
I applied it on the above sample and it does give the required transcription.