Introducing the Open Arabic LLM Leaderboard
โข
69
Ali El Filali
alielfilali01
AI & ML interests
"AI Psychometrician" | NLP (mainly for Arabic) | Other interests include Reinforcement Learning and Cognitive sciences among others
Articles
Organizations
alielfilali01's activity
reacted to
nroggendorff's
post with ๐
3 days ago
Post
2028
I still think whitespace in tokenizers are so dumb.
Congrats, you just doubled your vocab size for no reason.
Congrats, you just doubled your vocab size for no reason.
- 1 reply
reacted to
fdaudens's
post with โค๏ธโ
4 days ago
Post
2238
Just tested Argilla's new data annotation feature - it's a game changer for AI project quality.
Upload CSVs, work with published datasets, or improve existing ones directly on HuggingFace Hub. Setup took < 2 minutes, no code needed (see example below where I selected a dataset to classify tweets in categories).
Real world impact: Missing in Chicago won a Pulitzer using a similar approach - 200 volunteers labeled police misconduct files to train their model. That's the power of good data annotation.
Three immediate use cases I see:
- Build collaborative training sets with your community (surprisingly underused in AI journalism)
- Turn your website chatbot logs into high-quality fine-tuning data
- Compare generated vs published content (great for SEO headlines)
Works for solo projects or teams up to 100 people. All integrated with HuggingFace Hub for immediate model training.
Interesting to see tools like this making data quality more accessible. Data quality is the hidden driver of AI success that we don't talk about enough.
- Check out the blogpost: https://huggingface.co/blog/argilla-ui-hub
- And the quickstart guide: https://docs.argilla.io/latest/getting_started/quickstart/
Upload CSVs, work with published datasets, or improve existing ones directly on HuggingFace Hub. Setup took < 2 minutes, no code needed (see example below where I selected a dataset to classify tweets in categories).
Real world impact: Missing in Chicago won a Pulitzer using a similar approach - 200 volunteers labeled police misconduct files to train their model. That's the power of good data annotation.
Three immediate use cases I see:
- Build collaborative training sets with your community (surprisingly underused in AI journalism)
- Turn your website chatbot logs into high-quality fine-tuning data
- Compare generated vs published content (great for SEO headlines)
Works for solo projects or teams up to 100 people. All integrated with HuggingFace Hub for immediate model training.
Interesting to see tools like this making data quality more accessible. Data quality is the hidden driver of AI success that we don't talk about enough.
- Check out the blogpost: https://huggingface.co/blog/argilla-ui-hub
- And the quickstart guide: https://docs.argilla.io/latest/getting_started/quickstart/
reacted to
albertvillanova's
post with โค๏ธ๐
10 days ago
Post
2956
๐ Exciting update! You can now compare multiple models side-by-side with the Hugging Face Open LLM Comparator! ๐
open-llm-leaderboard/comparator
Dive into multi-model evaluations, pinpoint the best model for your needs, and explore insights across top open LLMs all in one place. Ready to level up your model comparison game?
open-llm-leaderboard/comparator
Dive into multi-model evaluations, pinpoint the best model for your needs, and explore insights across top open LLMs all in one place. Ready to level up your model comparison game?
reacted to
yjernite's
post with โค๏ธ
12 days ago
Post
๐ท๐ฝโโ๏ธ๐๐จ Announcing the Foundation Model Development Cheatsheet!
My first ๐คPost๐ค ever to announce the release of a fantastic collaborative resource to support model developers across the full development stack: The FM Development Cheatsheet available here: https://fmcheatsheet.org/
The cheatsheet is a growing database of the many crucial resources coming from open research and development efforts to support the responsible development of models. This new resource highlights essential yet often underutilized tools in order to make it as easy as possible for developers to adopt best practices, covering among other aspects:
๐ง๐ผโ๐คโ๐ง๐ผ data selection, curation, and governance;
๐ accurate and limitations-aware documentation;
โก energy efficiency throughout the training phase;
๐ thorough capability assessments and risk evaluations;
๐ environmentally and socially conscious deployment strategies.
We strongly encourage developers working on creating and improving models to make full use of the tools listed here, and to help keep the resource up to date by adding the resources that you yourself have developed or found useful in your own practice ๐ค
Congrats to all the participants in this effort for the release! Read more about it from:
@Shayne - https://twitter.com/ShayneRedford/status/1763215814860186005
@hails and @stellaathena - https://blog.eleuther.ai/fm-dev-cheatsheet/
@alon-albalak - http://nlp.cs.ucsb.edu/blog/a-new-guide-for-the-responsible-development-of-foundation-models.html
And also to @gabrielilharco @sayashk @kklyman @kylel @mbrauh @fauxneticien @avi-skowron @Bertievidgen Laura Weidinger, Arvind Narayanan, @VictorSanh @Davlan @percyliang Rishi Bommasani, @breakend @sasha ๐ฅ
My first ๐คPost๐ค ever to announce the release of a fantastic collaborative resource to support model developers across the full development stack: The FM Development Cheatsheet available here: https://fmcheatsheet.org/
The cheatsheet is a growing database of the many crucial resources coming from open research and development efforts to support the responsible development of models. This new resource highlights essential yet often underutilized tools in order to make it as easy as possible for developers to adopt best practices, covering among other aspects:
๐ง๐ผโ๐คโ๐ง๐ผ data selection, curation, and governance;
๐ accurate and limitations-aware documentation;
โก energy efficiency throughout the training phase;
๐ thorough capability assessments and risk evaluations;
๐ environmentally and socially conscious deployment strategies.
We strongly encourage developers working on creating and improving models to make full use of the tools listed here, and to help keep the resource up to date by adding the resources that you yourself have developed or found useful in your own practice ๐ค
Congrats to all the participants in this effort for the release! Read more about it from:
@Shayne - https://twitter.com/ShayneRedford/status/1763215814860186005
@hails and @stellaathena - https://blog.eleuther.ai/fm-dev-cheatsheet/
@alon-albalak - http://nlp.cs.ucsb.edu/blog/a-new-guide-for-the-responsible-development-of-foundation-models.html
And also to @gabrielilharco @sayashk @kklyman @kylel @mbrauh @fauxneticien @avi-skowron @Bertievidgen Laura Weidinger, Arvind Narayanan, @VictorSanh @Davlan @percyliang Rishi Bommasani, @breakend @sasha ๐ฅ
- 1 reply
reacted to
fdaudens's
post with โค๏ธ
15 days ago
Post
2737
๐คฏ Plot twist: Size isn't everything in AI! A lean 32B parameter model just showed up to the party and outperformed a 70B one. Efficiency > Scale? The AI world just got more interesting...
Cohere For AI released Aya Expanse, a new family of multilingual models (8B and 32B) spanning 23 popular languages.
Models: CohereForAI/c4ai-aya-expanse-671a83d6b2c07c692beab3c3
Blog post: https://huggingface.co/blog/aya-expanse
Demo: CohereForAI/aya_expanse
Cohere For AI released Aya Expanse, a new family of multilingual models (8B and 32B) spanning 23 popular languages.
Models: CohereForAI/c4ai-aya-expanse-671a83d6b2c07c692beab3c3
Blog post: https://huggingface.co/blog/aya-expanse
Demo: CohereForAI/aya_expanse
reacted to
clem's
post with ๐ฅ
15 days ago
Post
4019
This is no Woodstock AI but will be fun nonetheless haha. Iโll be hosting a live workshop with team members next week about the Enterprise Hugging Face hub.
1,000 spots available first-come first serve with some surprises during the stream!
You can register and add to your calendar here: https://streamyard.com/watch/JS2jHsUP3NDM
1,000 spots available first-come first serve with some surprises during the stream!
You can register and add to your calendar here: https://streamyard.com/watch/JS2jHsUP3NDM
- 4 replies
reacted to
abhishek's
post with ๐ค
19 days ago
replied to
their
post
19 days ago
I don't think i totally follow up what you are saying !?
posted
an
update
19 days ago
Post
1552
I feel like this incredible resource hasn't gotten the attention it deserves in the community!
@clefourrier and generally the HuggingFace evaluation team put together a fantastic guidebook covering a lot about ๐๐ฉ๐๐๐จ๐๐ง๐๐ข๐ก from basics to advanced tips.
link : https://github.com/huggingface/evaluation-guidebook
I havenโt finished it yet, but i'am enjoying every piece of it so far. Huge thanks @clefourrier and the team for this invaluable resource !
@clefourrier and generally the HuggingFace evaluation team put together a fantastic guidebook covering a lot about ๐๐ฉ๐๐๐จ๐๐ง๐๐ข๐ก from basics to advanced tips.
link : https://github.com/huggingface/evaluation-guidebook
I havenโt finished it yet, but i'am enjoying every piece of it so far. Huge thanks @clefourrier and the team for this invaluable resource !
-
-
- 3 replies
reacted to
mattmdjaga's
post with ๐ฅ
23 days ago
Post
1348
๐จ New Agent Benchmark ๐จ
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
ai-safety-institute/AgentHarm
Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.
The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.
We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2410.09024)
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents
ai-safety-institute/AgentHarm
Collaboration between UK AI Safety Institute and Gray Swan AI to create a dataset for measuring harmfulness of LLM agents.
The benchmark contains both harmful and benign sets of 11 categories with varied difficulty levels and detailed evaluation, not only testing success rate but also tool level accuracy.
We provide refusal and accuracy metrics across a wide range of models in both no attack and prompt attack scenarios.
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2410.09024)
reacted to
singhsidhukuldeep's
post with ๐
25 days ago
Post
1997
Just started going through the latest "State of AI Report 2024", and I cannot get over the predictions!
The report predicts major developments in AI over the next 12 months, including a $10B+ investment from a sovereign state into a large US AI lab, triggering national security scrutiny, and a viral app created by someone without coding skills.
It forecasts changes in data collection practices due to frontier labs facing trials, softer-than-expected EU AI Act implementations, and the rise of an open-source alternative to OpenAI GPT-4 outperforming in benchmarks.
NVIDIAโs dominance will remain largely unchallenged, investment in humanoid robots will decline, Appleโs on-device AI research will gain momentum, and a research paper by an AI scientist will be accepted at a major conference.
Lastly, a GenAI-based video game is expected to achieve breakout success.
Yet to go through all 200+ pages... will post summarized thoughts later.
The report predicts major developments in AI over the next 12 months, including a $10B+ investment from a sovereign state into a large US AI lab, triggering national security scrutiny, and a viral app created by someone without coding skills.
It forecasts changes in data collection practices due to frontier labs facing trials, softer-than-expected EU AI Act implementations, and the rise of an open-source alternative to OpenAI GPT-4 outperforming in benchmarks.
NVIDIAโs dominance will remain largely unchallenged, investment in humanoid robots will decline, Appleโs on-device AI research will gain momentum, and a research paper by an AI scientist will be accepted at a major conference.
Lastly, a GenAI-based video game is expected to achieve breakout success.
Yet to go through all 200+ pages... will post summarized thoughts later.
- 2 replies
reacted to
mervenoyan's
post with ๐ฅ
26 days ago
Post
2198
we have a leaderboard for video LLMs, and most of the top models are open ones!
opencompass/openvlm_video_leaderboard ๐๐
we are so back ๐ฅ
we are so back ๐ฅ
reacted to
clem's
post with โค๏ธ
27 days ago
Post
4128
Open-source AI creates healthy competition in a field where natural tendencies lead to extreme concentration of power. Imagine a world where only one or two companies could build software. This is the biggest risk and ethical challenge of them all IMO. Let's fight this!
- 3 replies
reacted to
merve's
post with ๐ฅ
27 days ago
Post
3698
Meta AI vision has been cooking
@facebook
They shipped multiple models and demos for their papers at @ECCV ๐ค
Here's a compilation of my top picks:
- Sapiens is family of foundation models for human-centric depth estimation, segmentation and more, all models have open weights and demos ๐
All models have their demos and even torchscript checkpoints!
A collection of models and demos: facebook/sapiens-66d22047daa6402d565cb2fc
- VFusion3D is state-of-the-art consistent 3D generation model from images
Model: facebook/vfusion3d
Demo: facebook/VFusion3D
- CoTracker is the state-of-the-art point (pixel) tracking model
Demo: facebook/cotracker
Model: facebook/cotracker
They shipped multiple models and demos for their papers at @ECCV ๐ค
Here's a compilation of my top picks:
- Sapiens is family of foundation models for human-centric depth estimation, segmentation and more, all models have open weights and demos ๐
All models have their demos and even torchscript checkpoints!
A collection of models and demos: facebook/sapiens-66d22047daa6402d565cb2fc
- VFusion3D is state-of-the-art consistent 3D generation model from images
Model: facebook/vfusion3d
Demo: facebook/VFusion3D
- CoTracker is the state-of-the-art point (pixel) tracking model
Demo: facebook/cotracker
Model: facebook/cotracker
reacted to
m-ric's
post with ๐ง โค๏ธ
27 days ago
Post
2239
๐ฅ ๐-๐๐ฎ๐ฅ: ๐๐๐๐ข๐ญ๐ข๐จ๐ง-๐๐ง๐ฅ๐ฒ ๐๐ฎ๐ฅ๐ญ๐ข๐ฉ๐ฅ๐ข๐๐๐ญ๐ข๐จ๐ง ๐๐๐ง ๐ฌ๐ฅ๐๐ฌ๐ก ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐๐ญ๐ข๐จ๐ง๐๐ฅ ๐๐จ๐ฌ๐ญ๐ฌ ๐๐ฒ ๐๐%!
Microsoft researchers dropped a groundbreaking technique that could slash the energy use in transformer computations : their novel "linear-complexity multiplication" (L-Mul) algorithm approximates floating-point multiplication using energy-efficient integer addition instead of costly multiplications.
๐ก Quick reminder on how floats are coded on 8 bits (FP8):
In the e4m3 FP8 standard, you encode a number as:
Sign (1 bit) | Exponent (4 bits) | Mantissa (3 bits)
Example: 0 (positive) | 1000 (8) | 101 (1/2 + 1/8 = 0.625)
Calculation: you add one to the mantissa, and multiply it by 2 power (the exponent - a bias term which is 7 for e4m3):
โก๏ธย You get (1 + 0.625) ร 2^(8-7) = 3.25
Now back to the paper. ๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
โก๏ธ Multiplication is extremely energy-intensive compared to addition. For 32-bit operations, multiplication (3.7 pJ) uses 37x more energy than addition (0.1 pJ)!
๐งฎ Traditional floating-point multiplication go like (noting xm the mantissa and xe the exponent): Mul(x,y) = (1 + xm) ยท 2^xe ยท (1 + ym) ยท 2^ye = (1 + xm + ym + xm ยท ym) ยท 2^(xe+ye)
๐ก L-Mul cleverly approximates this as: L-Mul(x,y) = (1 + xm + ym + 2^-l(m)) ยท 2^(xe+ye), eliminating the costly xm ยท ym term
๐ง l(m) term is adaptively set based on mantissa size for optimal accuracy
๐ Benchmarks on the Llama-3.1-8B-Instruct model show L-Mul preserves precision across various NLP tasks, with performance nearly identical to full BFloat16 precision
๐ฌ Authors claim: "We can achieve the same model inference performance while reducing the energy cost of attention computations by 80%."
This breakthrough is still theoretical and would need implementation on dedicated hardware to confirm real-world gains, but itโs a really exciting path for more sustainable AI! ๐ฑ
Read the paper here ๐ย Addition is All You Need for Energy-efficient Language Models (2410.00907)
Microsoft researchers dropped a groundbreaking technique that could slash the energy use in transformer computations : their novel "linear-complexity multiplication" (L-Mul) algorithm approximates floating-point multiplication using energy-efficient integer addition instead of costly multiplications.
๐ก Quick reminder on how floats are coded on 8 bits (FP8):
In the e4m3 FP8 standard, you encode a number as:
Sign (1 bit) | Exponent (4 bits) | Mantissa (3 bits)
Example: 0 (positive) | 1000 (8) | 101 (1/2 + 1/8 = 0.625)
Calculation: you add one to the mantissa, and multiply it by 2 power (the exponent - a bias term which is 7 for e4m3):
โก๏ธย You get (1 + 0.625) ร 2^(8-7) = 3.25
Now back to the paper. ๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
โก๏ธ Multiplication is extremely energy-intensive compared to addition. For 32-bit operations, multiplication (3.7 pJ) uses 37x more energy than addition (0.1 pJ)!
๐งฎ Traditional floating-point multiplication go like (noting xm the mantissa and xe the exponent): Mul(x,y) = (1 + xm) ยท 2^xe ยท (1 + ym) ยท 2^ye = (1 + xm + ym + xm ยท ym) ยท 2^(xe+ye)
๐ก L-Mul cleverly approximates this as: L-Mul(x,y) = (1 + xm + ym + 2^-l(m)) ยท 2^(xe+ye), eliminating the costly xm ยท ym term
๐ง l(m) term is adaptively set based on mantissa size for optimal accuracy
๐ Benchmarks on the Llama-3.1-8B-Instruct model show L-Mul preserves precision across various NLP tasks, with performance nearly identical to full BFloat16 precision
๐ฌ Authors claim: "We can achieve the same model inference performance while reducing the energy cost of attention computations by 80%."
This breakthrough is still theoretical and would need implementation on dedicated hardware to confirm real-world gains, but itโs a really exciting path for more sustainable AI! ๐ฑ
Read the paper here ๐ย Addition is All You Need for Energy-efficient Language Models (2410.00907)
reacted to
victor's
post with โโค๏ธ
27 days ago
Post
2500
NEW - Inference Playground
Maybe like me you have always wanted a super easy way to compare llama3.2-1B vs. llama3.2-3B? or the same model with different temperatures?
Trying and comparing warm Inference API models has never been easier!
Just go to https://hf.co/playground, set your token and you're ready to go.
We'll keep improving, feedback welcome ๐
Maybe like me you have always wanted a super easy way to compare llama3.2-1B vs. llama3.2-3B? or the same model with different temperatures?
Trying and comparing warm Inference API models has never been easier!
Just go to https://hf.co/playground, set your token and you're ready to go.
We'll keep improving, feedback welcome ๐
- 2 replies