Xenova/larger_clap_general

https://huggingface.co/laion/larger_clap_general with ONNX weights to be compatible with Transformers.js.

Usage (Transformers.js)

If you haven't already, you can install the Transformers.js JavaScript library from NPM using:

npm i @xenova/transformers

Example: Perform zero-shot audio classification with Xenova/larger_clap_general.

import { pipeline } from '@xenova/transformers';

const classifier = await pipeline('zero-shot-audio-classification', 'Xenova/larger_clap_general');

const audio = 'https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/piano.wav';
const candidate_labels = ['calm piano music', 'heavy metal music'];
const scores = await classifier(audio, candidate_labels);
// [
//   { score: 0.9829504489898682, label: 'calm piano music' },
//   { score: 0.017049523070454597, label: 'heavy metal music' }
// ]

Example: Compute text embeddings with ClapTextModelWithProjection.

import { AutoTokenizer, ClapTextModelWithProjection } from '@xenova/transformers';

// Load tokenizer and text model
const tokenizer = await AutoTokenizer.from_pretrained('Xenova/larger_clap_general');
const text_model = await ClapTextModelWithProjection.from_pretrained('Xenova/larger_clap_general');

// Run tokenization
const texts = ['calm piano music', 'heavy metal music'];
const text_inputs = tokenizer(texts, { padding: true, truncation: true });

// Compute embeddings
const { text_embeds } = await text_model(text_inputs);
// Tensor {
//   dims: [ 2, 512 ],
//   type: 'float32',
//   data: Float32Array(1024) [ ... ],
//   size: 1024
// }

Example: Compute audio embeddings with ClapAudioModelWithProjection.

import { AutoProcessor, ClapAudioModelWithProjection, read_audio } from '@xenova/transformers';

// Load processor and audio model
const processor = await AutoProcessor.from_pretrained('Xenova/larger_clap_general');
const audio_model = await ClapAudioModelWithProjection.from_pretrained('Xenova/larger_clap_general');

// Read audio and run processor
const audio = await read_audio('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/piano.wav');
const audio_inputs = await processor(audio);

// Compute embeddings
const { audio_embeds } = await audio_model(audio_inputs);
// Tensor {
//   dims: [ 1, 512 ],
//   type: 'float32',
//   data: Float32Array(512) [ ... ],
//   size: 512
// }

---

Note: Having a separate repo for ONNX weights is intended to be a temporary solution until WebML gains more traction. If you would like to make your models web-ready, we recommend converting to ONNX using 🤗 Optimum and structuring your repo like this one (with ONNX weights located in a subfolder named onnx).