Update README.md

README.md

@@ -2800,4 +2800,214 @@ model-index:
     - type: v_measure
       value: 85.30624598674467
 license: apache-2.0
----
+---
+<h1 align="center">Snowflake's artic-embed-m</h1>
+<h4 align="center">
+   <p>
+       <a href=#news>News</a> |
+       <a href=#models>Models</a> |
+       <a href=#usage>Usage</a>  |
+       <a href="#evaluation">Evaluation</a> |
+       <a href="#contact">Contact</a> |
+       <a href="#faq">FAQ</a>
+       <a href="#license">License</a> |
+       <a href="#acknowledgement">Acknowledgement</a>
+   <p>
+</h4>
+
+
+## News
+
+
+04/16/2024: Release the ** Arctic-text-embed ** family of text empedding models. The releases are state-of-the-art for Retrieval quality at each of their representative size profiles. [Technical Report]() is coming shortly. For more details, please refer to our Github: [Arctic-Text-Embed](https://github.com/Snowflake/Arctic-Text-Embed).
+
+
+## Models
+
+
+Arctic-Embed is a suite of text embedding models that focuses on creating high-quality embedding models for retrieval that are optimized for performance.
+
+
+The `arctic-text-embedding` models achieve **state-of-the-art performance on the MTEB/BEIR leaderboard** for each of their size variants. Evaluation is performed using these [scripts](https://github.com/Snowflake-Labs/arctic-embed/tree/main/src). As shown below, each class of model size achieves SOTA retrieval accuracy when compared to other top models.
+
+
+The models are trained by leveraging existing open-source text representation models, such as bert-base-uncased, and are trained in a multi-stage pipeline to optimize their retrieval performance. First, the models are trained with large batches of query-document pairs where negatives are derived in-batch—pretraining leverages about 400m samples of a mix of public datasets and proprietary web search data. Following pretraining models are further optimized with long training on a smaller dataset (about 1m samples) of triplets of query, positive document, and negative document derived from hard harmful mining. Mining of the negatives and data curation is crucial to retrieval accuracy. A detailed technical report will be available shortly.
+
+
+| Name                                                                    | MTEB Retrieval Score (NDCG @ 10) | Parameters (Millions) | Embedding Dimension |
+| ----------------------------------------------------------------------- | -------------------------------- | --------------------- | ------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-xs/)     | 50.15                            | 22                    | 384                 |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-s/)      | 51.98                            | 33                    | 384                 |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m/)      | 54.90                            | 110                   | 768                 |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m-long/) | 54.83                            | 137                   | 768                 |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-l/)      | 55.98                            | 335                   | 1024                |
+
+
+Aside from being great open-source models, the largest model, [arctic-embed-l](https://huggingface.co/Snowflake/arctic-embed-l/), can serve as a natural replacement for closed-source embedding, as shown below.
+
+
+| Model Name                                                         | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------ | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-l/) | 55.98                            |
+| Google-gecko-text-embedding                                        | 55.7                             |
+| text-embedding-3-large                                             | 55.44                            |
+| Cohere-embed-english-v3.0                                          | 55.00                            |
+| bge-large-en-v1.5                                                  | 54.29                            |
+
+
+### [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-xs/)
+
+
+This tiny model packs quite the punch based on the [all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model. With only 22m parameters and 384 dimensions, this model should meet even the strictest latency/TCO budgets. Despite its size, its retrieval accuracy is closer to that of models with 100m paramers.
+
+
+| Model Name                                                          | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------- | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-xs/) | 50.15                            |
+| GIST-all-MiniLM-L6-v2                                               | 45.12                            |
+| gte-tiny                                                            | 44.92                            |
+| all-MiniLM-L6-v2                                                    | 41.95                            |
+| bge-micro-v2                                                        | 42.56                            |
+
+
+### Arctic-embed-m
+
+
+Based on the [all-MiniLM-L12-v2](https://huggingface.co/intfloat/e5-base-unsupervised) model, this small model does not trade off retrieval accuracy for its small size. With only 33m parameters and 384 dimensions, this model should easily allow scaling to large datasets.
+
+
+| Model Name                                                         | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------ | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-s/) | 51.98                            |
+| bge-small-en-v1.5                                                  | 51.68                            |
+| Cohere-embed-english-light-v3.0                                    | 51.34                            |
+| text-embedding-3-small                                             | 51.08                            |
+| e5-small-v2                                                        | 49.04                            |
+
+
+### [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m-long/)
+
+
+Based on the [nomic-embed-text-v1](https://huggingface.co/nomic-ai/nomic-embed-text-v1) model, this long-context variant of our medium-sized model is perfect for workloads that can be constrained by the regular 512 token context of our other models. Without the use of RPE, this model supports up to 2048 tokens. With RPE, it can scale to 8192!
+
+
+| Model Name                                                         | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------ | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m/) | 54.90                            |
+| bge-base-en-v1.5                                                   | 53.25                            |
+| nomic-embed-text-v1.5                                              | 53.01                            |
+| GIST-Embedding-v0                                                  | 52.31                            |
+| gte-base                                                           | 52.31                            |
+
+
+### Arctic-embed-m
+
+
+Based on the [intfloat/e5-base-unsupervised](https://huggingface.co/intfloat/e5-base-unsupervised) model, this medium model is the workhorse that provides the best retrieval performance without slowing down inference.
+
+
+| Model Name                                                         | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------ | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m/) | 54.90                            |
+| bge-base-en-v1.5                                                   | 53.25                            |
+| nomic-embed-text-v1.5                                              | 53.25                            |
+| GIST-Embedding-v0                                                  | 52.31                            |
+| gte-base                                                           | 52.31                            |
+
+
+### [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-l/)
+
+
+Based on the [intfloat/e5-large-unsupervised](https://huggingface.co/intfloat/e5-large-unsupervised) model, this small model does not sacrifice retrieval accuracy for its small size.
+
+
+| Model Name                                                         | MTEB Retrieval Score (NDCG @ 10) |
+| ------------------------------------------------------------------ | -------------------------------- |
+| [arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-l/) | 55.98                            |
+| UAE-Large-V1                                                       | 54.66                            |
+| bge-large-en-v1.5                                                  | 54.29                            |
+| mxbai-embed-large-v1                                               | 54.39                            |
+| e5-Large-v2                                                        | 50.56                            |
+
+
+## Usage
+
+
+### Using Huggingface transformers
+
+
+To use an arctic-embed model, you can use the transformers package, as shown below. For optimal retrieval quality, ensure that you use the CLS token as the embedding for each portion of text and use the query prefix below (just on the query).
+
+
+
+```python
+import torch
+from transformers import AutoModel, AutoTokenizer
+
+tokenizer = AutoTokenizer.from_pretrained('Snowflake/snow-text-embed-base')
+model = AutoModel.from_pretrained('Snowflake/snow-text-embed-base', add_pooling_layer=False)
+model.eval()
+
+query_prefix = 'Represent this sentence for searching relevant passages: '
+queries  = ['what is snowflake?', 'Where can I get the best tacos?']
+queries_with_prefix = ["{}{}".format(query_prefix, i) for i in queries]
+query_tokens = tokenizer(queries_with_prefix, padding=True, truncation=True, return_tensors='pt', max_length=512)
+
+documents = ['The Data Cloud!', 'Mexico City of Course!']
+document_tokens =  tokenizer(documents, padding=True, truncation=True, return_tensors='pt', max_length=512)
+
+# Compute token embeddings
+with torch.no_grad():
+    query_embeddings = model(**query_tokens)[0][:, 0]
+    doument_embeddings = model(**document_tokens)[0][:, 0]
+
+
+# normalize embeddings
+query_embeddings = torch.nn.functional.normalize(query_embeddings, p=2, dim=1)
+doument_embeddings = torch.nn.functional.normalize(doument_embeddings, p=2, dim=1)
+
+scores = torch.mm(query_embeddings, doument_embeddings.transpose(0, 1))
+for query, query_scores in zip(queries, scores):
+    doc_score_pairs = list(zip(documents, query_scores))
+    doc_score_pairs = sorted(doc_score_pairs, key=lambda x: x[1], reverse=True)
+    #Output passages & scores
+    print("Query:", query)
+    for document, score in doc_score_pairs:
+        print(score, document)
+```
+
+
+If you use the long context model and have more than 2048 tokens, ensure that you initialize the model like below instead. This will use [RPE](https://arxiv.org/abs/2104.09864) to allow up to 8192 tokens.
+
+
+``` py
+model = AutoModel.from_pretrained('Snowflake/[arctic-embed-s](https://huggingface.co/Snowflake/arctic-embed-m-long/)', trust_remote_code=True, rotary_scaling_factor=2)
+```
+
+
+## FAQ
+
+
+TBD
+
+
+## Contact
+
+
+If you have any questions or suggestions about  this project, feel free to open an issue or pull request.
+You also can email Daniel Campos([email protected]).
+
+
+## License
+
+
+Arctic is licensed under the [Apache-2](https://www.apache.org/licenses/LICENSE-2.0). The released models can be used for commercial purposes free of charge.
+
+
+## Acknowledgement
+
+
+We would like to thank the open-source community, which has provided the great building blocks upon which we could make our models.
+
+
+