New collection needs to be looked at. Some numbers arent adding up.

#995
by rombodawg - opened

So I noticed that one of my models was not being recognized on this new collection. Im assuming this collection was made automatically with code, and there is most likely an error in the logic.

https://huggingface.co/collections/open-llm-leaderboard/open-llm-leaderboard-best-models-652d6c7965a4619fb5c27a03

As you can see the "Around 13b" models my 14b outperforms the "failspy/Phi-3-medium-4k-instruct-abliterated-v3" model. Not that im trying to be a leaderboard hog or anything, but you want to have an accurate collection, otherwise people are being misinformed about the data.

Screenshot (932).png

Screenshot (931).png

I am curious. Is this because the qwen-14b models are 14.8b params? And the collection is only picking up, lets say, up to 14.1b params?

Open LLM Leaderboard org

Not that im trying to be a leaderboard hog or anything, but you want to have an accurate collection, otherwise people are being misinformed about the data.

We really like when users share these kind of issues with us, as it allows us to make the leaderboard better for everyone - as long as you're polite we're super glad to get feedback like this! So don't worry, and thanks for your vigilance!

Open LLM Leaderboard org

Hi @rombodawg ,

Thanks for reporting! Yes, it's an automatic process to form this collection, I revised this code a little bit so the collection should better represent the best models now. Please, check it out here – link

Feel free to share your thoughts!

@alozowski I just checked it out. It looked way better. more organized too. Good job 👍👍

Open LLM Leaderboard org

Thank you! Really appreciate your help 🤝

Feel free to open a discussion if you have any issues or suggestions!

alozowski changed discussion status to closed

Sign up or log in to comment