Spaces:

open-llm-leaderboard

/

open_llm_leaderboard

like 11.8k

Running on CPU Upgrade

@clefourrier Can you take a look at this when you have some free time. Thanks.

Hi!
At a first glance, this looks nice! If I understood it properly, it creates corresponding PRs on results and requests for a given model name change?
If yes, you would also need to change the details repository to match the new model name.
But overall, looks like nice work, I'll likely give it a try next week!

Hi

Thanks for you comments.

I examined your commits to rename a model in the leaderboard to make this app but I missed the details repository. Will work on details repo too. You can see the example if you want here:

• https://huggingface.co/datasets/Weyaxi/requests/discussions/21/files

• https://huggingface.co/datasets/Weyaxi/results/discussions/29/files

I found something interesting. Last week, I requested to rename a model, and it was renamed perfectly. However, I am seeing the old name in the results dataset. Link:

• https://huggingface.co/datasets/open-llm-leaderboard/details_PulsarAI__MythoMax-L2-LoRA-Assemble-13B

@Weyaxi I'm quite sure I know where this comes from but we won't be able to fix it before the end of the month (we have something big brewing behind the scene, but no spoilers!)
Keep it in mind or open an issue somewhere else so we can track it and take care of it once we're good on our side.

@clefourrier , even though the details repo's name may not be crucial for me (since I can see the correct model name on the front), it is still something to consider. I can rename the config in the details repo, but I don't think I can open a pull request to rename the entire repository. I'm pretty sure there are a lot of things going on behind the scenes, so good luck with all your work.

So how should I proceed in this situation?

Thanks.

Already changing the config in the detail repo and all its names should be sufficient, if we get all the PRs open we'll manually change it. It would be nice to add a reminder to change the repo name from one to another in the PR text if you really want to go all the way.
Looking forward to this :)

No problem in my side Will be working on this.

@clefourrier An example:

• https://huggingface.co/datasets/Weyaxi/requests/discussions/34
• https://huggingface.co/datasets/Weyaxi/results/discussions/42
• https://huggingface.co/datasets/open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch/discussions/2

Super cool! Last change, would it be possible to change the title of the PR from "Renaming models" to "Renaming model X to Y" so that it's easier to follow that they all are for the same rename?
@SaylorTwift do you have the time to take a look here to tell me if there's any other renaming place we missed?

Thanks, the title of the PR's are now changed. Links:

• https://huggingface.co/datasets/open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch/discussions/3/files

• https://huggingface.co/datasets/Weyaxi/results/discussions/43/files

• https://huggingface.co/datasets/Weyaxi/requests/discussions/35/files

This looks great!

You just need to change the targets of the results and requests to be the actual open-llm-leaderboard repos and I'll tweet it, and add it to the FAQ and resources tab!
Oh and update the name in the README of the details file if possible!
Great work, you're going to help us and users win a lot of time :)

@clefourrier Okay, I have just updated the space to match the actual datasets (instead of using my cloned ones). The first PR is as follows:

• https://huggingface.co/datasets/open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch/discussions/4

• https://huggingface.co/datasets/open-llm-leaderboard/requests/discussions/14

• https://huggingface.co/datasets/open-llm-leaderboard/results/discussions/8

If you have any requests about the app, I will be happy to help :)

I will try to change the readme too. Sorry I missed that.

Hi @clefourrier , I have updated the space. Now, the README is changing, but I haven't changed the links (changed the model link but not the details repo's link in README) in the details repo because the repo name itself remains the same. Here is an example:

• https://huggingface.co/datasets/open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch/discussions/6/files

• https://huggingface.co/datasets/open-llm-leaderboard/results/discussions/10

• https://huggingface.co/datasets/open-llm-leaderboard/requests/discussions/16/files

Do you mean in this section?

from datasets import load_dataset
data = load_dataset("open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch",
    "harness_truthfulqa_mc_0",
    split="train")

You can change it as we'll move the details repo manually to fit the new names

Yes I meant that section. Okay I will change that as well.

@clefourrierI think I did it? Is everything okay?

• https://huggingface.co/datasets/open-llm-leaderboard/requests/discussions/19
• https://huggingface.co/datasets/open-llm-leaderboard/results/discussions/13
• https://huggingface.co/datasets/open-llm-leaderboard/details_TFLai__OpenOrca-Platypus2-13B-QLoRA-0.80-epoch/discussions/9

This looks amazing, congrats!!!

Thanks :)

I added your tool to the FAQ and Resources tab, and I tweeted about it here!
If you want to add a bit more doc to your space (especially for beginner users) that would be very nice, but it is usable as such!
Tysm!

Thank you for your comments :). I can add some documentation in the future.

On a different note, would it be a good idea to create a discord webhook app to track models on the leaderboard?

I don't know if there is a need for this on discord, as I personally don't use it, so I'll tag @osanseviero who will likely know the answer to this :)
A tracker in general could be useful however, in the form of a space for example?

The transition from Discord to X doesn't seem to be quite difficult, but how will users be informed?

really really cool stuff @Weyaxi 🔥

Thank you so much @julien-c :)

cc @lunarflu re: Discord, lots of cool stuff!

@clefourrier Can this be closed, since it's already done?

Hi @jensdraht ,
I left it open so people get access to the resource easily, since it's not used a lot at the moment. I'll add it in the FAQ then close it.