Love the UGI Leaderboard. We should add a quantized category - as not all quantizations are equal
I really like the UGI leaderboard. However, if models are being tested quantized, I would posit that it would be a good idea to have a quantized category, so we can see how each model performs under every quantization submitted, as not all quantizations are equal - this will affect the model’s writing, thinking, and, its other capabilities. Just food for thought. 😋
Update: I read the UGI LeaderBoard’s page: models are tested in Q4_K_M that’s a good idea. Having one quant to quantize all models to is a great way to ensure every model gets an equal shot at the UGI Leaderboard. 🤖
I think your update resolves this? If you're asking for links to the used quants, I recorded a lot but not all of the ones I used, so I'd kinda rather not make it a column.