intfloat

/

multilingual-e5-large-instruct

like 234

I am reading papers about the E5 family and decoder based embeddings. References are:

https://arxiv.org/pdf/2402.05672.pdf
https://arxiv.org/pdf/2212.03533.pdf
https://arxiv.org/pdf/2401.00368.pdf

Regarding the instruction based finetunning E5 family I think I understand how training is performed. We have a siamese network that shares weights. We input an instruction + query to obtain an embedding, and the document to obtain another embeding. And then minimize a constrative loss.

However, from the FAQs I see we need to provide instructions when generating embeddings from downstream tasks: https://huggingface.co/intfloat/multilingual-e5-large-instruct#faq

However, from my understanding and what I see in the code this is not necessary. My points are:

1. If the model generates an embedding from instruction + query and another for document ( see section 4.1 at https://arxiv.org/pdf/2212.03533.pdf) for constrative loss, then generating the embedding for the document is not really informed by the instruction+query. Model weights are, but not the embedding generation.

2. Following previous point if we look at the code example at huggingface https://huggingface.co/intfloat/multilingual-e5-large-instruct#transformers , we see a concatenation of queries + documents resulting in a batch of 4 samples (first two samples are english and chinese query and final two samples their corresponding documents). This generates 4 embeddings which are not being informed by other samples in the batch. Embeddings are just informed from the tokens of the sample in the batch per se. So this should mean that generating an embedding for query + documents should result in a similar embedding for, e.g. query, then if only query is passed through the model.

3. I have checked point 2 and its true. Here is the output of two runs of the HF code.

The embedding obtained for the input:

Instruct: Given a web search query, retrieve relevant passages that answer the query \n QUery: how much protein should a female eat

Has values:

0.020, 0.0112, -0.0451...

If I input both the instruct + query and document, i.e.:
Instruct: Given a web search query, retrieve relevant passages that answer the query \n QUery: how much protein should a female eat
As a general guidelin, the CDC's average requirement of protein for woman ages 19 to 70 is 48 grams ....

I obtain two embeddings, the first one with exactly the same values

0.020, 0.0112, -0.0451...
0.0327, 0.0041, -0.0503...

So in conclusion, what is this FAQ being refered to? I can understand that is necessary for training but not for obtaining embeddings for downstream applications.