Spaces:
Running
How to request performance evaluation with 'llm-perf-backend'?
I am planning to submit LLM models to this leaderboard for testing various compression and quantization frameworks.
I saw the description "Hardware/Backend/Optimization performance requests should be made in the llm-perf-backend repository and will be added to the 🤗 LLM-Perf Leaderboard 🏋️ automatically."
Which script(s) do I need to run for performance evaluation requests in 'llm-perf-backend'? With a quick tour of scripts, maybe the benchmark script and push-dataset script seems enough. However, I want to confirm the community's opinion in advance.
Hi @LLMNewbie, we will soon refactor the LLM Perf Leaderboard to benchmark a list of canonical LLMs, instead of scrapping a list of unique architectures from the Open LLM Leaderboard. We can discuss model submissions then but generally a quantized version of a finetuned version of a pretrained model will have the same hardware performance as the the quantized pretrained model.
@IlyasMoutawwakil Thanks for the info. Looking forward to submit & test the models on this ldbd. It's hard to find leaderboards that explore the efficiency of LLM. :(