College biology isn't medicine

#4
No description provided.
maximegmd changed pull request title from College biology isn't medecine to College biology isn't medicine
Open Life Science AI org

Hi @maximegmd , we also consider biology as part of the evaluation as it "evaluates" biomedical knowledge. We would like to evaluate the biomedical knowledge of the LLMs, not only medicine/clinical. Furthermore, MultiMedQA suggested by the Med-PaLM paper (Singhal et al., 2023) also includes MMLU college biology.

Let us know if you disagree, we would love to hear your take!

Hello,

I do not believe it evaluates biomedical knowledge. I went through the questions and the questions are mostly irrelevant (questions about animal behavior, geology, plants...).

While multimedqa did include it, they are in my opinion mistaken and it shows that involving medical doctors in these decisions is necessary.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment