Hugging Face's logo

Hub documentation

Appendix

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Appendix

Appendix A: User Study

Full text responses to key questions

How would you define model cards?

Insight: Respondents had generally similar views of what model cards are: documentation focused on issues like training, use cases, and bias/limitations

  • Model cards are model descriptions, both of how they were trained, their use cases, and potential biases and limitations
  • Documents describing the essential features of a model in order for the reader/user to understand the artefact he/she has in front, the background/training, how it can be used, and its technical/ethical limitations.
  • They serve as a living artefact of models to document them. Model cards contain information that go from a high level description of what the specific model can be used to, to limitations, biases, metrics, and much more. They are used primarily to understand what the model does.
  • Model cards are to models what GitHub READMEs are to GitHub projects. It tells people all the information they need to know about the model. If you don’t write one, nobody will use your model.
  • From what I understand, a model card uses certain benchmarks (geography, culture, sex, etc) to define both a model’s usability and limitations. It’s essentially a model’s ‘nutrition facts label’ that can show how a model was created and educates others on its reusability.
  • Model cards are the metadata and documentation about the model, everything I need to know to use the model properly: info about the model, what paper introduced it, what dataset was it trained on or fine-tuned on, whom does it belong to, are there known risks and limitations with this model, any useful technical info.
  • IMO model cards are a brief presentation of a model which includes:
    • short summary of the architectural particularities of the model
    • describing the data it was trained on
    • what is the performance on reference datasets (accuracy and speed metrics if possible)
    • limitations
    • how to use it in the context of the Transformers library
    • source (original article, Github repo,…)
  • Easily accessible documentation that any background can read and learn about critical model components and social impact

What do you like about model cards?

  • They are interesting to teach people about new models
  • As a non-technical guy, the possibility of getting to know the model, to understand the basics of it, it’s an opportunity for the author to disclose its innovation in a transparent & explainable (i.e. trustworthy) way.
  • I like interactive model cards with visuals and widgets that allow me to try the model without running any code.
  • What I like about good model cards is that you can find all the information you need about that particular model.
  • Model cards are revolutionary to the world of AI ethics. It’s one of the first tangible steps in mitigating/educating on biases in machine learning. They foster greater awareness and accountability!
  • Structured, exhaustive, the more info the better.
  • It helps to get an understanding of what the model is good (or bad) at.
  • Conciseness and accessibility

What do you dislike about model cards?

  • Might get to technical and/or dense
  • They contain lots of information for different audiences (researchers, engineers, non engineers), so it's difficult to explore model cards with an intended use cases. * [NOTE: this comment could be addressed with toggle views for different audiences]
  • Good ones are time consuming to create. They are hard to test to make sure the information is up to date. Often times, model cards are formatted completely differently - so you have to sort of figure out how that certain individual has structured theirs. * [NOTE: this comment helps demonstrate the value of a standardized format and automation tools to make it easier to create model cards]
  • Without the help of the community to pitch in supplemental evals, model cards might be subject to inherent biases that the developer might not be aware of. It’s early days for them, but without more thorough evaluations, a model card’s information might be too limited.
  • Empty model cards. No license information - customers need that info and generally don't have it.
  • They are usually either too concise or too verbose.
  • writing them lol bless you

Other key new insights

  • Model cards are best filled out when done by people with different roles: Technical specifications can generally only be filled out by the developers; ethical considerations throughout are generally best informed by people who tend to work on ethical issues.
  • Model users care a lot about licences — specifically, whether a model can legally be used for a specific task.

Appendix B: Landscape Analysis

Overview of the state of model documentation in Machine Learning

MODEL CARD EXAMPLES

Examples of model cards and closely-related variants include:

MODEL CARDS FOR LARGE LANGUAGE MODELS

Large language models are often released with associated documentation. Large language models that have an associated model card (or related documentation tool) include:

MODEL CARD GENERATION TOOLS

Tools for programmatically or interactively generating model cards include:

MODEL CARD EDUCATIONAL TOOLS

Tools for understanding model cards and understanding how to create model cards include:

Please cite as: Ozoani, Ezi and Gerchick, Marissa and Mitchell, Margaret. Model Card Guidebook. Hugging Face, 2022. https://huggingface.co/docs/hub/en/model-card-guidebook

< > Update on GitHub

🤗 Hugging Face Hub→