Hugging Face's logo

tomaarsen
/
distilbert-base-uncased-sts-matryoshka

Sentence Similarity
sentence-transformers
Safetensors
English
distilbert
feature-extraction
loss:MatryoshkaLoss
loss:CoSENTLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
Edit model card

SentenceTransformer based on distilbert/distilbert-base-uncased

This is a sentence-transformers model finetuned from distilbert/distilbert-base-uncased on the sentence-transformers/stsb dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: DistilBertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/distilbert-base-uncased-sts-matryoshka")
# Run inference
sentences = [
    'A woman is dancing.',
    'A woman is dancing in railway station.',
    'The flag was moving in the air.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

Metric Value
pearson_cosine 0.8648
spearman_cosine 0.8748
pearson_manhattan 0.8628
spearman_manhattan 0.8658
pearson_euclidean 0.8627
spearman_euclidean 0.8658
pearson_dot 0.7443
spearman_dot 0.7514
pearson_max 0.8648
spearman_max 0.8748

Semantic Similarity

Metric Value
pearson_cosine 0.8628
spearman_cosine 0.8741
pearson_manhattan 0.862
spearman_manhattan 0.8651
pearson_euclidean 0.8623
spearman_euclidean 0.8653
pearson_dot 0.7464
spearman_dot 0.7541
pearson_max 0.8628
spearman_max 0.8741

Semantic Similarity

Metric Value
pearson_cosine 0.8589
spearman_cosine 0.8714
pearson_manhattan 0.8591
spearman_manhattan 0.8634
pearson_euclidean 0.8592
spearman_euclidean 0.8629
pearson_dot 0.7186
spearman_dot 0.7289
pearson_max 0.8592
spearman_max 0.8714

Semantic Similarity

Metric Value
pearson_cosine 0.8529
spearman_cosine 0.8688
pearson_manhattan 0.8509
spearman_manhattan 0.8576
pearson_euclidean 0.8532
spearman_euclidean 0.8581
pearson_dot 0.697
spearman_dot 0.7059
pearson_max 0.8532
spearman_max 0.8688

Semantic Similarity

Metric Value
pearson_cosine 0.834
spearman_cosine 0.8587
pearson_manhattan 0.8352
spearman_manhattan 0.8446
pearson_euclidean 0.8387
spearman_euclidean 0.8461
pearson_dot 0.6579
spearman_dot 0.6713
pearson_max 0.8387
spearman_max 0.8587

Semantic Similarity

Metric Value
pearson_cosine 0.8337
spearman_cosine 0.847
pearson_manhattan 0.8485
spearman_manhattan 0.847
pearson_euclidean 0.8493
spearman_euclidean 0.8475
pearson_dot 0.6702
spearman_dot 0.6526
pearson_max 0.8493
spearman_max 0.8475

Semantic Similarity

Metric Value
pearson_cosine 0.8326
spearman_cosine 0.8468
pearson_manhattan 0.8474
spearman_manhattan 0.8463
pearson_euclidean 0.8482
spearman_euclidean 0.8466
pearson_dot 0.6737
spearman_dot 0.6572
pearson_max 0.8482
spearman_max 0.8468

Semantic Similarity

Metric Value
pearson_cosine 0.8226
spearman_cosine 0.8403
pearson_manhattan 0.8421
spearman_manhattan 0.842
pearson_euclidean 0.8435
spearman_euclidean 0.8429
pearson_dot 0.623
spearman_dot 0.6062
pearson_max 0.8435
spearman_max 0.8429

Semantic Similarity

Metric Value
pearson_cosine 0.815
spearman_cosine 0.835
pearson_manhattan 0.8352
spearman_manhattan 0.8361
pearson_euclidean 0.8376
spearman_euclidean 0.8376
pearson_dot 0.5958
spearman_dot 0.5793
pearson_max 0.8376
spearman_max 0.8376

Semantic Similarity

Metric Value
pearson_cosine 0.7981
spearman_cosine 0.827
pearson_manhattan 0.8239
spearman_manhattan 0.8289
pearson_euclidean 0.8279
spearman_euclidean 0.8315
pearson_dot 0.5206
spearman_dot 0.5067
pearson_max 0.8279
spearman_max 0.8315

Training Details

Training Dataset

sentence-transformers/stsb

  • Dataset: sentence-transformers/stsb at ab7a5ac
  • Size: 5,749 training samples
  • Columns: sentence1, sentence2, and score
  • Approximate statistics based on the first 1000 samples:
    sentence1 sentence2 score
    type string string float
    details
    • min: 6 tokens
    • mean: 10.0 tokens
    • max: 28 tokens
    • min: 5 tokens
    • mean: 9.95 tokens
    • max: 25 tokens
    • min: 0.0
    • mean: 0.54
    • max: 1.0
  • Samples:
    sentence1 sentence2 score
    A plane is taking off. An air plane is taking off. 1.0
    A man is playing a large flute. A man is playing a flute. 0.76
    A man is spreading shreded cheese on a pizza. A man is spreading shredded cheese on an uncooked pizza. 0.76
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "CoSENTLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Evaluation Dataset

sentence-transformers/stsb

  • Dataset: sentence-transformers/stsb at ab7a5ac
  • Size: 1,500 evaluation samples
  • Columns: sentence1, sentence2, and score
  • Approximate statistics based on the first 1000 samples:
    sentence1 sentence2 score
    type string string float
    details
    • min: 5 tokens
    • mean: 15.1 tokens
    • max: 45 tokens
    • min: 6 tokens
    • mean: 15.11 tokens
    • max: 53 tokens
    • min: 0.0
    • mean: 0.47
    • max: 1.0
  • Samples:
    sentence1 sentence2 score
    A man with a hard hat is dancing. A man wearing a hard hat is dancing. 1.0
    A young child is riding a horse. A child is riding a horse. 0.95
    A man is feeding a mouse to a snake. The man is feeding a mouse to the snake. 1.0
  • Loss: MatryoshkaLoss with these parameters:
    {
    "loss": "CoSENTLoss",
    "matryoshka_dims": [
        768,
        512,
        256,
        128,
        64
    ],
    "matryoshka_weights": [
        1,
        1,
        1,
        1,
        1
    ],
    "n_dims_per_step": -1
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • num_train_epochs: 4
  • warmup_ratio: 0.1
  • fp16: True

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: False
  • per_device_train_batch_size: 16
  • per_device_eval_batch_size: 16
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: None
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss sts-dev-128_spearman_cosine sts-dev-256_spearman_cosine sts-dev-512_spearman_cosine sts-dev-64_spearman_cosine sts-dev-768_spearman_cosine sts-test-128_spearman_cosine sts-test-256_spearman_cosine sts-test-512_spearman_cosine sts-test-64_spearman_cosine sts-test-768_spearman_cosine
0.2778 100 23.266 21.5517 0.8305 0.8355 0.8361 0.8157 0.8366 - - - - -
0.5556 200 21.8736 21.6172 0.8327 0.8388 0.8446 0.8206 0.8453 - - - - -
0.8333 300 21.6241 22.0565 0.8475 0.8538 0.8556 0.8345 0.8565 - - - - -
1.1111 400 21.075 23.6719 0.8545 0.8581 0.8634 0.8435 0.8644 - - - - -
1.3889 500 20.4122 22.5926 0.8592 0.8624 0.8650 0.8436 0.8656 - - - - -
1.6667 600 20.6586 22.5999 0.8514 0.8563 0.8595 0.8389 0.8597 - - - - -
1.9444 700 20.3262 22.2965 0.8582 0.8631 0.8666 0.8465 0.8667 - - - - -
2.2222 800 19.7948 23.1844 0.8621 0.8659 0.8688 0.8499 0.8694 - - - - -
2.5 900 19.2826 23.1351 0.8653 0.8687 0.8703 0.8547 0.8710 - - - - -
2.7778 1000 19.1063 23.7141 0.8641 0.8672 0.8691 0.8531 0.8695 - - - - -
3.0556 1100 19.4575 23.0055 0.8673 0.8702 0.8726 0.8574 0.8728 - - - - -
3.3333 1200 18.0727 24.9288 0.8659 0.8692 0.8715 0.8565 0.8722 - - - - -
3.6111 1300 18.1698 25.3114 0.8675 0.8701 0.8728 0.8576 0.8734 - - - - -
3.8889 1400 18.2321 25.3777 0.8688 0.8714 0.8741 0.8587 0.8748 - - - - -
4.0 1440 - - - - - - - 0.8350 0.8403 0.8468 0.8270 0.8470

Environmental Impact

Carbon emissions were measured using CodeCarbon.

  • Energy Consumed: 0.020 kWh
  • Carbon Emitted: 0.008 kg of CO2
  • Hours Used: 0.112 hours

Training Hardware

  • On Cloud: No
  • GPU Model: 1 x NVIDIA GeForce RTX 3090
  • CPU Model: 13th Gen Intel(R) Core(TM) i7-13700K
  • RAM Size: 31.78 GB

Framework Versions

  • Python: 3.11.6
  • Sentence Transformers: 3.0.0.dev0
  • Transformers: 4.41.0.dev0
  • PyTorch: 2.3.0+cu121
  • Accelerate: 0.26.1
  • Datasets: 2.18.0
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss 

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

CoSENTLoss 

@online{kexuefm-8847,
    title={CoSENT: A more efficient sentence vector scheme than Sentence-BERT},
    author={Su Jianlin},
    year={2022},
    month={Jan},
    url={https://kexue.fm/archives/8847},
}
Downloads last month
4
Safetensors
Model size
66.4M params
Tensor type
F32
·
Inference Examples
Sentence Similarity
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Model tree for tomaarsen/distilbert-base-uncased-sts-matryoshka

Base model

distilbert/distilbert-base-uncased
Finetuned
(6582)
this model

Evaluation results

View on Papers With Code