--- language: - zh model-index: - name: Chuxin-Embedding results: - dataset: config: default name: MTEB CmedqaRetrieval (default) revision: cd540c506dae1cf9e9a59c3e06f42030d54e7301 split: dev type: C-MTEB/CmedqaRetrieval metrics: - type: map_at_1 value: 33.391999999999996 - type: map_at_10 value: 48.715 - type: map_at_100 value: 50.381 - type: map_at_1000 value: 50.456 - type: map_at_3 value: 43.708999999999996 - type: map_at_5 value: 46.405 - type: mrr_at_1 value: 48.612 - type: mrr_at_10 value: 58.67099999999999 - type: mrr_at_100 value: 59.38 - type: mrr_at_1000 value: 59.396 - type: mrr_at_3 value: 55.906 - type: mrr_at_5 value: 57.421 - type: ndcg_at_1 value: 48.612 - type: ndcg_at_10 value: 56.581 - type: ndcg_at_100 value: 62.422999999999995 - type: ndcg_at_1000 value: 63.476 - type: ndcg_at_3 value: 50.271 - type: ndcg_at_5 value: 52.79899999999999 - type: precision_at_1 value: 48.612 - type: precision_at_10 value: 11.995000000000001 - type: precision_at_100 value: 1.696 - type: precision_at_1000 value: 0.185 - type: precision_at_3 value: 27.465 - type: precision_at_5 value: 19.675 - type: recall_at_1 value: 33.391999999999996 - type: recall_at_10 value: 69.87100000000001 - type: recall_at_100 value: 93.078 - type: recall_at_1000 value: 99.55199999999999 - type: recall_at_3 value: 50.939 - type: recall_at_5 value: 58.714 - type: main_score value: 56.581 task: type: Retrieval - dataset: config: default name: MTEB CovidRetrieval (default) revision: 1271c7809071a13532e05f25fb53511ffce77117 split: dev type: C-MTEB/CovidRetrieval metrics: - type: map_at_1 value: 71.918 - type: map_at_10 value: 80.609 - type: map_at_100 value: 80.796 - type: map_at_1000 value: 80.798 - type: map_at_3 value: 79.224 - type: map_at_5 value: 79.96 - type: mrr_at_1 value: 72.076 - type: mrr_at_10 value: 80.61399999999999 - type: mrr_at_100 value: 80.801 - type: mrr_at_1000 value: 80.803 - type: mrr_at_3 value: 79.276 - type: mrr_at_5 value: 80.025 - type: ndcg_at_1 value: 72.076 - type: ndcg_at_10 value: 84.286 - type: ndcg_at_100 value: 85.14500000000001 - type: ndcg_at_1000 value: 85.21 - type: ndcg_at_3 value: 81.45400000000001 - type: ndcg_at_5 value: 82.781 - type: precision_at_1 value: 72.076 - type: precision_at_10 value: 9.663 - type: precision_at_100 value: 1.005 - type: precision_at_1000 value: 0.101 - type: precision_at_3 value: 29.398999999999997 - type: precision_at_5 value: 18.335 - type: recall_at_1 value: 71.918 - type: recall_at_10 value: 95.574 - type: recall_at_100 value: 99.473 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 87.82900000000001 - type: recall_at_5 value: 90.991 - type: main_score value: 84.286 task: type: Retrieval - dataset: config: default name: MTEB DuRetrieval (default) revision: a1a333e290fe30b10f3f56498e3a0d911a693ced split: dev type: C-MTEB/DuRetrieval metrics: - type: map_at_1 value: 25.019999999999996 - type: map_at_10 value: 77.744 - type: map_at_100 value: 80.562 - type: map_at_1000 value: 80.60300000000001 - type: map_at_3 value: 52.642999999999994 - type: map_at_5 value: 67.179 - type: mrr_at_1 value: 86.5 - type: mrr_at_10 value: 91.024 - type: mrr_at_100 value: 91.09 - type: mrr_at_1000 value: 91.093 - type: mrr_at_3 value: 90.558 - type: mrr_at_5 value: 90.913 - type: ndcg_at_1 value: 86.5 - type: ndcg_at_10 value: 85.651 - type: ndcg_at_100 value: 88.504 - type: ndcg_at_1000 value: 88.887 - type: ndcg_at_3 value: 82.707 - type: ndcg_at_5 value: 82.596 - type: precision_at_1 value: 86.5 - type: precision_at_10 value: 41.595 - type: precision_at_100 value: 4.7940000000000005 - type: precision_at_1000 value: 0.48900000000000005 - type: precision_at_3 value: 74.233 - type: precision_at_5 value: 63.68000000000001 - type: recall_at_1 value: 25.019999999999996 - type: recall_at_10 value: 88.114 - type: recall_at_100 value: 97.442 - type: recall_at_1000 value: 99.39099999999999 - type: recall_at_3 value: 55.397 - type: recall_at_5 value: 73.095 - type: main_score value: 85.651 task: type: Retrieval - dataset: config: default name: MTEB EcomRetrieval (default) revision: 687de13dc7294d6fd9be10c6945f9e8fec8166b9 split: dev type: C-MTEB/EcomRetrieval metrics: - type: map_at_1 value: 55.60000000000001 - type: map_at_10 value: 67.891 - type: map_at_100 value: 68.28699999999999 - type: map_at_1000 value: 68.28699999999999 - type: map_at_3 value: 64.86699999999999 - type: map_at_5 value: 66.652 - type: mrr_at_1 value: 55.60000000000001 - type: mrr_at_10 value: 67.891 - type: mrr_at_100 value: 68.28699999999999 - type: mrr_at_1000 value: 68.28699999999999 - type: mrr_at_3 value: 64.86699999999999 - type: mrr_at_5 value: 66.652 - type: ndcg_at_1 value: 55.60000000000001 - type: ndcg_at_10 value: 74.01100000000001 - type: ndcg_at_100 value: 75.602 - type: ndcg_at_1000 value: 75.602 - type: ndcg_at_3 value: 67.833 - type: ndcg_at_5 value: 71.005 - type: precision_at_1 value: 55.60000000000001 - type: precision_at_10 value: 9.33 - type: precision_at_100 value: 1.0 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 25.467000000000002 - type: precision_at_5 value: 16.8 - type: recall_at_1 value: 55.60000000000001 - type: recall_at_10 value: 93.30000000000001 - type: recall_at_100 value: 100.0 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 76.4 - type: recall_at_5 value: 84.0 - type: main_score value: 74.01100000000001 task: type: Retrieval - dataset: config: default name: MTEB MMarcoRetrieval (default) revision: 539bbde593d947e2a124ba72651aafc09eb33fc2 split: dev type: C-MTEB/MMarcoRetrieval metrics: - type: map_at_1 value: 66.24799999999999 - type: map_at_10 value: 75.356 - type: map_at_100 value: 75.653 - type: map_at_1000 value: 75.664 - type: map_at_3 value: 73.515 - type: map_at_5 value: 74.67099999999999 - type: mrr_at_1 value: 68.496 - type: mrr_at_10 value: 75.91499999999999 - type: mrr_at_100 value: 76.17399999999999 - type: mrr_at_1000 value: 76.184 - type: mrr_at_3 value: 74.315 - type: mrr_at_5 value: 75.313 - type: ndcg_at_1 value: 68.496 - type: ndcg_at_10 value: 79.065 - type: ndcg_at_100 value: 80.417 - type: ndcg_at_1000 value: 80.72399999999999 - type: ndcg_at_3 value: 75.551 - type: ndcg_at_5 value: 77.505 - type: precision_at_1 value: 68.496 - type: precision_at_10 value: 9.563 - type: precision_at_100 value: 1.024 - type: precision_at_1000 value: 0.105 - type: precision_at_3 value: 28.391 - type: precision_at_5 value: 18.086 - type: recall_at_1 value: 66.24799999999999 - type: recall_at_10 value: 89.97 - type: recall_at_100 value: 96.13199999999999 - type: recall_at_1000 value: 98.551 - type: recall_at_3 value: 80.624 - type: recall_at_5 value: 85.271 - type: main_score value: 79.065 task: type: Retrieval - dataset: config: default name: MTEB MedicalRetrieval (default) revision: 2039188fb5800a9803ba5048df7b76e6fb151fc6 split: dev type: C-MTEB/MedicalRetrieval metrics: - type: map_at_1 value: 61.8 - type: map_at_10 value: 71.101 - type: map_at_100 value: 71.576 - type: map_at_1000 value: 71.583 - type: map_at_3 value: 68.867 - type: map_at_5 value: 70.272 - type: mrr_at_1 value: 61.9 - type: mrr_at_10 value: 71.158 - type: mrr_at_100 value: 71.625 - type: mrr_at_1000 value: 71.631 - type: mrr_at_3 value: 68.917 - type: mrr_at_5 value: 70.317 - type: ndcg_at_1 value: 61.8 - type: ndcg_at_10 value: 75.624 - type: ndcg_at_100 value: 77.702 - type: ndcg_at_1000 value: 77.836 - type: ndcg_at_3 value: 71.114 - type: ndcg_at_5 value: 73.636 - type: precision_at_1 value: 61.8 - type: precision_at_10 value: 8.98 - type: precision_at_100 value: 0.9900000000000001 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 25.867 - type: precision_at_5 value: 16.74 - type: recall_at_1 value: 61.8 - type: recall_at_10 value: 89.8 - type: recall_at_100 value: 99.0 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 77.60000000000001 - type: recall_at_5 value: 83.7 - type: main_score value: 75.624 task: type: Retrieval - dataset: config: default name: MTEB T2Retrieval (default) revision: 8731a845f1bf500a4f111cf1070785c793d10e64 split: dev type: C-MTEB/T2Retrieval metrics: - type: map_at_1 value: 27.173000000000002 - type: map_at_10 value: 76.454 - type: map_at_100 value: 80.021 - type: map_at_1000 value: 80.092 - type: map_at_3 value: 53.876999999999995 - type: map_at_5 value: 66.122 - type: mrr_at_1 value: 89.519 - type: mrr_at_10 value: 92.091 - type: mrr_at_100 value: 92.179 - type: mrr_at_1000 value: 92.183 - type: mrr_at_3 value: 91.655 - type: mrr_at_5 value: 91.94 - type: ndcg_at_1 value: 89.519 - type: ndcg_at_10 value: 84.043 - type: ndcg_at_100 value: 87.60900000000001 - type: ndcg_at_1000 value: 88.32799999999999 - type: ndcg_at_3 value: 85.623 - type: ndcg_at_5 value: 84.111 - type: precision_at_1 value: 89.519 - type: precision_at_10 value: 41.760000000000005 - type: precision_at_100 value: 4.982 - type: precision_at_1000 value: 0.515 - type: precision_at_3 value: 74.944 - type: precision_at_5 value: 62.705999999999996 - type: recall_at_1 value: 27.173000000000002 - type: recall_at_10 value: 82.878 - type: recall_at_100 value: 94.527 - type: recall_at_1000 value: 98.24199999999999 - type: recall_at_3 value: 55.589 - type: recall_at_5 value: 69.476 - type: main_score value: 84.043 task: type: Retrieval - dataset: config: default name: MTEB VideoRetrieval (default) revision: 58c2597a5943a2ba48f4668c3b90d796283c5639 split: dev type: C-MTEB/VideoRetrieval metrics: - type: map_at_1 value: 70.1 - type: map_at_10 value: 79.62 - type: map_at_100 value: 79.804 - type: map_at_1000 value: 79.804 - type: map_at_3 value: 77.81700000000001 - type: map_at_5 value: 79.037 - type: mrr_at_1 value: 70.1 - type: mrr_at_10 value: 79.62 - type: mrr_at_100 value: 79.804 - type: mrr_at_1000 value: 79.804 - type: mrr_at_3 value: 77.81700000000001 - type: mrr_at_5 value: 79.037 - type: ndcg_at_1 value: 70.1 - type: ndcg_at_10 value: 83.83500000000001 - type: ndcg_at_100 value: 84.584 - type: ndcg_at_1000 value: 84.584 - type: ndcg_at_3 value: 80.282 - type: ndcg_at_5 value: 82.472 - type: precision_at_1 value: 70.1 - type: precision_at_10 value: 9.68 - type: precision_at_100 value: 1.0 - type: precision_at_1000 value: 0.1 - type: precision_at_3 value: 29.133 - type: precision_at_5 value: 18.54 - type: recall_at_1 value: 70.1 - type: recall_at_10 value: 96.8 - type: recall_at_100 value: 100.0 - type: recall_at_1000 value: 100.0 - type: recall_at_3 value: 87.4 - type: recall_at_5 value: 92.7 - type: main_score value: 83.83500000000001 task: type: Retrieval tags: - mteb --- # Chuxin-Embedding Chuxin-Embedding 是专为增强中文文本检索能力而设计的嵌入模型。它基于bge-m3-retromae[1]，实现了预训练、微调、精调全流程。该模型在来自各个领域的大量语料库上进行训练，语料库的批量非常大。截至 2024 年 9 月 14 日, Chuxin-Embedding 在检索任务中表现出色，在 C-MTEB 中文检索排行榜上排名第一，领先的性能得分为 77.88，在AIR-Bench中文检索+重排序公开排行榜上排名第一，领先的性能得分为 64.78。 Chuxin-Embedding is a specially designed embedding model aimed at enhancing the capability of Chinese text retrieval. It is based on bge-m3-retromae[1] and implements the entire process of pre-training, fine-tuning, and refining. This model has been trained on a vast amount of corpora from various fields. As of September 14, 2024, Chuxin-Embedding has shown outstanding performance in retrieval tasks. It ranks first on the C-MTEB Chinese Retrieval Leaderboard with a leading performance score of 77.88 and also ranks first on the AIR-Bench Chinese Retrieval + Re-ranking Public Leaderboard with a leading performance score of 64.78. ## News - 2024/10/18: LLM生成及数据清洗 [Code](https://github.com/chuxin-llm/Chuxin-Embedding/blob/main/README_LLM.md) 。 - 2024/9/14: 团队的RAG框架欢迎试用 [ragnify](https://github.com/chuxin-llm/ragnify) 。 - 2024/9/14: LLM generation and data clean [Code](https://github.com/chuxin-llm/Chuxin-Embedding) . - 2024/9/14: The team's RAG framework is available for trial [ragnify](https://github.com/chuxin-llm/ragnify) . ## Training Details ![image/png](chuxinembedding.png) 基于bge-m3-retromae[1]，主要改动如下： - 基于bge-m3-retromae[1]在亿级数据上预训练。 - 使用BGE pretrain [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain) 完成预训练。 - 在收集的公开亿级检索数据集上实现了微调。 - 使用BGE finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) 完成微调。 - 在收集的公开百万级检索数据集和百万级LLM合成数据集上实现了精调。 - 使用BGE finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) 和 BGE unified_finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune) 完成精调。 - 通过 LLM (QWEN-72B) 进行数据生成，使用 LLM 为message生成新query - 数据清洗： - 简单的基于规则清洗 - LLM判断是否可作为搜索引擎查询的query - rerank模型对(query,message)评分，舍弃pos中的负例，neg中的正例 Based on bge-m3-retromae[1], the main modifications are as follows: - Pre-trained on a billion-level dataset based on bge-m3-retromae[1]. - Pre-training is completed using BGE pretrain [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/pretrain) . - Fine-tuned on a publicly collected billion-level retrieval dataset. - Fine-tuning is completed using BGE finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune). - Refined on a publicly collected million-level retrieval dataset and a million-level LLM synthetic dataset. - Refining is completed using BGE finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/finetune) and BGE unified_finetune [Code](https://github.com/FlagOpen/FlagEmbedding/tree/master/examples/unified_finetune). - Data generation is performed through LLM (QWEN-72B), using LLM to generate new query for messages. - Data cleaning: - Simple rule-based cleaning - LLM to determine whether a query can be used as a search engine query - The rerank model scores (query, message) pairs, discarding negative examples in the positive set and positive examples in the negative set. ## Collect more data for retrieval-type tasks 1. 预训练数据 - ChineseWebText、 oasis、 oscar、 SkyPile、 wudao 2. 微调数据 - MTP 、webqa、nlpcc、csl、bq、atec、ccks 3. 精调数据 - BGE-M3 、Huatuo26M-Lite 、covid ... - LLM 合成（BGE-M3 、Huatuo26M-Lite 、covid、wudao、wanjuan_news、mnbvc_news_wiki、mldr、medical QA...） ## Performance **C_MTEB RETRIEVAL** | Model | **Average** | **CmedqaRetrieval** | **CovidRetrieval** | **DuRetrieval** | **EcomRetrieval** | **MedicalRetrieval** | **MMarcoRetrieval** | **T2Retrieval** | **VideoRetrieval** | | :-------------------: | :---------: | :-------: | :------------: | :-----------: | :-----------: | :-------: | :----------: | :-------: | :----------: | | Zhihui_LLM_Embedding | 76.74 | 48.69 | 84.39 | 91.34 | 71.96 | 65.19 | 84.77 |88.3 | 79.31 | | zpoint_large_embedding_zh | 76.36 | 47.16 | 89.14 | 89.23 | 70.74 | 68.14 | 82.38 | 83.81 | 80.26 | | **Chuxin-Embedding** | **77.88** | 56.58 | 84.28 | 85.65 | 74.01 | 75.62 | 79.06 | 84.04 | 83.84 | **AIR-Bench** | Retrieval Method | Reranking Model | **Average** | **wiki_zh** | **web_zh** | **news_zh** | **healthcare_zh** | **finance_zh** | | :-------------------: | :---------:| :---------: | :-------: | :------------: | :-----------: | :-----------: | :----------: | | bge-m3 | bge-reranker-large | 64.53 | 76.11 | 67.8 | 63.25 | 62.9 | 52.61 | | gte-Qwen2-7B-instruct |bge-reranker-large | 63.39 | 78.09 | 67.56 | 63.14 | 61.12 | 47.02 | | **Chuxin-Embedding** | bge-reranker-large | **64.78** |76.23 | 68.44 | 64.2 | 62.93 | 52.11 | ## Generate Embedding for text ```python #pip install -U FlagEmbedding from FlagEmbedding import FlagModel model = FlagModel('chuxin-llm/Chuxin-Embedding', query_instruction_for_retrieval="为这个句子生成表示以用于检索相关文章：", use_fp16=True) sentences_1 = ["样例数据-1", "样例数据-2"] sentences_2 = ["样例数据-3", "样例数据-1"] embeddings_1 = model.encode(sentences_1) embeddings_2 = model.encode(sentences_2) similarity = embeddings_1 @ embeddings_2.T print(similarity) ``` ### Reference 1. https://huggingface.co/BAAI/bge-m3-retromae 2. https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/BGE_M3 3. https://github.com/FlagOpen/FlagEmbedding/tree/master/FlagEmbedding/baai_general_embedding