Xenova/larger_clap_music_and_speech

https://huggingface.co/laion/larger_clap_music_and_speech with ONNX weights to be compatible with Transformers.js.

Usage (Transformers.js)

If you haven't already, you can install the Transformers.js JavaScript library from NPM using:

npm i @xenova/transformers

Example: Perform zero-shot audio classification with Xenova/larger_clap_music_and_speech.

import { pipeline } from '@xenova/transformers';

const classifier = await pipeline('zero-shot-audio-classification', 'Xenova/larger_clap_music_and_speech');

const audio = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav';
const candidate_labels = ['mlk speech', 'upbeat pop song'];
const scores = await classifier(audio, candidate_labels);
// [
//   { score: 0.9970206022262573, label: 'mlk speech' },
//   { score: 0.002979411045089364, label: 'upbeat pop song' }
// ]

Example: Compute text embeddings with ClapTextModelWithProjection.

import { AutoTokenizer, ClapTextModelWithProjection } from '@xenova/transformers';

// Load tokenizer and text model
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/larger_clap_music_and_speech');
const text_model = await ClapTextModelWithProjection.from_pretrained('Xenova/larger_clap_music_and_speech');

// Run tokenization
const texts = ['an upbeat pop song', 'a passionate speech'];
const text_inputs = tokenizer(texts, { padding: true, truncation: true });

// Compute embeddings
const { text_embeds } = await text_model(text_inputs);
// Tensor {
//   dims: [ 2, 512 ],
//   type: 'float32',
//   data: Float32Array(1024) [ ... ],
//   size: 1024
// }

Example: Compute audio embeddings with ClapAudioModelWithProjection.

import { AutoProcessor, ClapAudioModelWithProjection, read_audio } from '@xenova/transformers';

// Load processor and audio model
const processor = await AutoProcessor.from_pretrained('Xenova/larger_clap_music_and_speech');
const audio_model = await ClapAudioModelWithProjection.from_pretrained('Xenova/larger_clap_music_and_speech');

// Read audio and run processor
const audio = await read_audio('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/mlk.wav');
const audio_inputs = await processor(audio);

// Compute embeddings
const { audio_embeds } = await audio_model(audio_inputs);
// Tensor {
//   dims: [ 1, 512 ],
//   type: 'float32',
//   data: Float32Array(512) [ ... ],
//   size: 512
// }

---

Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).