openlifescienceai/open_medical_llm_leaderboard · Add model `johnsnowlabs/JSL-MedMNX-7B` to medical leaderboard.

Apr 15

Hi, We at John Snow Labs have created a model johnsnowlabs/JSL-MedMNX-7B which performs great on medical benchmarks. It appears that we cannot submit our model at the moment. Please add the following model to the leaderboard as it outperforms Nexusflow/Starling-LM-7B-beta.

Model Name: johnsnowlabs/JSL-MedMNX-7B
Evaluation:

Tasks	Version	Filter	Metric	Value		Stderr
stem	N/A	none	acc_norm	0.5191	±	0.0068
		none	acc	0.5658	±	0.0058
- medmcqa	Yaml	none	acc	0.5135	±	0.0077
		none	acc_norm	0.5135	±	0.0077
- medqa_4options	Yaml	none	acc	0.5373	±	0.0140
		none	acc_norm	0.5373	±	0.0140
- anatomy (mmlu)	0	none	acc	0.6370	±	0.0415
- clinical_knowledge (mmlu)	0	none	acc	0.7245	±	0.0275
- college_biology (mmlu)	0	none	acc	0.7500	±	0.0362
- college_medicine (mmlu)	0	none	acc	0.6590	±	0.0361
- medical_genetics (mmlu)	0	none	acc	0.7200	±	0.0451
- professional_medicine (mmlu)	0	none	acc	0.7206	±	0.0273
- pubmedqa	1	none	acc	0.7720	±	0.0188

Groups	Version	Filter	n-shot	Metric	Value		Stderr
stem	N/A	none	0	acc_norm	0.5191	±	0.0068
		none	0	acc	0.5658	±	0.0058

aryopg

Open Life Science AI org Apr 15

Hi @abideen , we're currently working on the backend to introduce new GPUs. Sorry for the delay, but hopefully the submitted models will be evaluated soon.

aryopg

Open Life Science AI org Apr 16

Hi @abideen , your model has been added to the leaderboard. I'll close this issue now 🙂

aryopg changed discussion status to closed Apr 16