This master model card serves as an entry point for exploring multiple speech-to-phoneme models trained on different sensor data from the Vibravox dataset.
These models are designed to convert French speech into sequences of International Phonetic Alphabet (IPA) encoded words, and are fine-tuned on specific sensors to address various audio capture scenarios using body conducted sound and vibration sensors.
Each of these models has been trained for specific non-conventional speech sensors and is intended to be used with in-domain data. The only exception is the headset microphone phonemizer, which can certainly be used for many applications using audio data captured by airborne microphones.
Please be advised that using these models outside their intended sensor data may result in suboptimal performance.
The primary task for these models is an ASR task in the speech-to-phoneme context. Each model takes audio input and outputs a sequence of phonemes encoded in the IPA, facilitating precise phonetic transcription of French speech. Users unfamiliar with the phonetic alphabet can use tools like the IPA reader to convert the transcript back to synthetic speech and evaluate the transcription quality.
All models are finetuned versions of facebook/wav2vec2-base-fr-voxpopuli-v2 and adapted to different sensor inputs. They are intended to be used at a sample rate of 16kHz.
The models were each finetuned for 10 epochs with a constant learning rate of 1e-5. Detailed instructions for reproducing the experiments are available on the jhauret/vibravox Github repository and in the VibraVox paper on arXiV.
The following models are available, each trained on a different sensor on the speech_clean subset of (https://huggingface.co/datasets/Cnam-LMSSC/vibravox):
| Transducer | Huggingface model link |
|---|---|
| Reference headset microphone | phonemizer_headset_microphone |
| In-ear comply foam-embedded microphone | phonemizer_soft_in_ear_microphone |
| In-ear rigid earpiece-embedded microphone | phonemizer_rigid_in_ear_microphone |
| Forehead miniature vibration sensor | phonemizer_forehead_accelerometer |
| Temple vibration pickup | phonemizer_temple_vibration_pickup |
| Laryngophone | phonemizer_throat_microphone |
