Supradeepdan/OverlapSpeakerCounter

Speaker Counting with XVector and ECAPA-TDNN models

This repository provides tools for detecting and counting the number of speakers in an audio recording using models like XVector and ECAPA-TDNN, trained on datasets such as LibriSpeech.

The pre-trained system processes audio inputs to detect the presence and count of speakers, providing output in the following format:

0.00-2.50 has 1 speaker
2.50-4.20 has 2 speakers
4.20-5:30 has no speakers

The system expects input recordings sampled at 16kHz. If your signal has a different sample rate, resample it using torchaudio before using the interface.

Model Performance

Error rates for different numbers of speakers detected by the models are provided below:

XVector Model Error Rates

Number of Speakers	Error Rate
No Speakers	0.00e+00
1 Speaker	4.06e-01
2 Speakers	3.20e-02
3 Speakers	2.15e-01
4 Speakers	4.67e-01
Overall	2.29e-01

ECAPA-TDNN Model Error Rates

Number of Speakers	Error Rate
No Speakers	0.00e+00
1 Speaker	4.12e-01
2 Speakers	4.10e-01
3 Speakers	3.26e-01
4 Speakers	5.23e-01
Overall	2.40e-01

Self-supervised MLP Model Error Rates

Number of Speakers	Error Rate
No Speakers	0.00e+00
1 Speaker	8.67e-03
2 Speakers	1.14e-01
3 Speakers	4.08e-01
4 Speakers	4.49e-01
Overall	2.00e-01

Self-supervised XVector Model Error Rates

Number of Speakers	Error Rate
No Speakers	0.00e+00
1 Speaker	1.64e-02
2 Speakers	1.39e-01
3 Speakers	3.34e-01
4 Speakers	5.41e-01
Overall	2.10e-01

Error rates are crucial to understanding where the models excel and where they may need further improvement. In particular, they can reveal challenges in scenarios with higher numbers of speakers.

Limitations

Please note that these models are evaluated under controlled conditions and their performance may vary when applied to different datasets or in real-world scenarios.

System Overview

This project includes interfaces for processing audio files, detecting speech, and counting speakers. It leverages pre-trained models and custom scripts for refining and aggregating predictions.

Installation

pip install speechbrain

Please look at SpeechBrain for tutorials or more information.

Using the Speaker Counter Interface

For XVector & ECAPA-TDNN:

from interface.SpeakerCounter import SpeakerCounter

wav_path = "path/to/your/audio.wav"
model_path = "path/to/your/model"
save_dir = "path/to/save/results"

# Initialize the SpeakerCounter
speaker_counter = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir)

# Run inference
speaker_counter.classify_file(wav_path)

For a Self-supervised model:

from interface.SpeakerCounterSelfsupervisedMLP import SpeakerCounter

wav_path = "path/to/your/audio.wav"
model_path = "path/to/your/selfsupervised_model"
save_dir = "path/to/save/results"

# Initialize the SpeakerCounter
audio_classifier = SpeakerCounter.from_hparams(source=model_path, savedir=save_dir)

# Run inference
audio_classifier.classify_file(wav_path)

Setup and Training Instructions

To reproduce the results of this project, follow these steps.

!git clone https://github.com/SupradeepDanturti/ConvAIProject
%cd ConvAIProject

Download the project code from the GitHub repository and navigate into the project directory.

!python prepare_dataset/download_required_data.py --output_folder <destination_folder_path>

Download the necessary datasets (LibriSpeech etc.), specifying the desired destination folder.

!python prepare_dataset/create_custom_dataset.py prepare_dataset/dataset.yaml

Create custom dataset based on set parameters as shown in the sample below

Sample of dataset.yaml:

n_sessions:
  train: 1000 # Creates 1000 sessions per class
  dev: 200 # Creates 200 sessions per class
  eval: 200 # Creates 200 sessions per class
n_speakers: 4 # max number of speakers. In this case the total classes will be 5 (0-4 speakers)
max_length: 120 # max length in seconds for each session/utterance.

Sample of dataset.yaml

To train the XVector model run the following command.

%cd xvector
!python train_xvector_augmentation.py hparams_xvector_augmentation.yaml

To train the ECAPA-TDNN model run the following command.

%cd ecapa_tdnn
!python train_ecapa_tdnn.py hparams_ecapa_tdnn_augmentation.yaml

To train the SelfSupervised MLP model run the following command.

%cd selfsupervised
!python selfsupervised_mlp.py hparams_selfsupervised_mlp.yaml

To train the SelfSupervised XVector model run the following command.

%cd selfsupervised
!python selfsupervised_xvector.py hparams_selfsupervised_xvector.yaml

**This Project was developed completely using SpeechBrain **

Reference

@misc{speechbrain,
  title={{SpeechBrain}: A General-Purpose Speech Toolkit},
  author={Mirco Ravanelli and Titouan Parcollet and Peter Plantinga and Aku Rouhe and Samuele Cornell and Loren Lugosch and Cem Subakan and Nauman Dawalatabad and Abdelwahab Heba and Jianyuan Zhong and Ju-Chieh Chou and Sung-Lin Yeh and Szu-Wei Fu and Chien-Feng Liao and Elena Rastorgueva and François Grondin and William Aris and Hwidong Na and Yan Gao and Renato De Mori and Yoshua Bengio},
  year={2021},
  eprint={2106.04624},
  archivePrefix={arXiv},
  primaryClass={eess.AS},
  note={arXiv:2106.04624}
}