Spaces:
Runtime error
Runtime error
How does the space know whether a model is fine-tuned or not?
#3
by
patrickvonplaten
- opened
I tried evaluated microsoft/deberta-v3 on GLUE, but it didn't show up in the list. This makes sense I guess since the model is only pretrained and not fine-tuned. How does the space know which models are fine-tuned and which aren't?
Hey @patrickvonplaten , the list of compatible models is determined by two criteria:
- Whether the
pipeline_tagin the model card matches the selected task - Whether the selected dataset belongs to one of the
datasetslisted on the model card
So yes, you won't find fill-mask models in the list right now as we don't support this (yet) in the backend - do you see a good use case for evaluating pretrained models?
For references, here's the filter I apply on the models: https://huggingface.co/spaces/autoevaluate/model-evaluator/blob/main/utils.py#L89