Our Transformers Code Agent beats the GAIA benchmark!
โข
44
Aymeric Roucher
m-ric
AI & ML interests
MLE at Hugging Face ๐ค
LLMs, Agents, RAG, Multimodal.
Articles
Organizations
m-ric's activity
posted
an
update
1 day ago
Post
1086
๐๐ฟ๐ฒ ๐๐ด๐ฒ๐ป๐๐ ๐ฐ๐ฎ๐ฝ๐ฎ๐ฏ๐น๐ฒ ๐ฒ๐ป๐ผ๐๐ด๐ต ๐ณ๐ผ๐ฟ ๐๐ฎ๐๐ฎ ๐ฆ๐ฐ๐ถ๐ฒ๐ป๐ฐ๐ฒ? โ ๐ ๐ฒ๐ฎ๐๐๐ฟ๐ฒ ๐๐ต๐ฒ๐ถ๐ฟ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐๐ถ๐๐ต ๐๐ฆ๐๐ฒ๐ป๐ฐ๐ต ๐
A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluationโฆ
โก๏ธ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
๐ง๐ต๐ฒ ๐๐ฆ๐๐ฒ๐ป๐ฐ๐ต ๐ฑ๐ฎ๐๐ฎ๐๐ฒ๐
โช๏ธDS bench has 466 data analysis tasks and 74 data modelling tasks
โช๏ธThe tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
โช๏ธDifference with previous DS benchmarks:
โถ This benchmark leverages various modalities on top of text: images, Excel files, tables
โท Complex tables: sometimes several tables should be leveraged to answer one question
โธ The context is richer, with longer descriptions.
โช๏ธ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
๐๐ป๐๐ถ๐ด๐ต๐๐ ๐ณ๐ฟ๐ผ๐บ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ป๐ด ๐ฎ๐ด๐ฒ๐ป๐๐
โช๏ธ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
โช๏ธ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
This new benchmark is really welcome, can't wait to try transformers agents on it! ๐ค
Read their full paper ๐ DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? (2409.07703)
A team from Tencent AI wanted to evaluate agentic systems on data science (DS) tasks : but they noticed that existing agentic benchmarks were severely limited in several aspects: they were limited to text and did not include tables or images, were only specific to certain packages, only performed exact match evaluationโฆ
โก๏ธ So they set out to build a much more exhaustive approach, to finally make the definitive DS agent benchmark.
๐ง๐ต๐ฒ ๐๐ฆ๐๐ฒ๐ป๐ฐ๐ต ๐ฑ๐ฎ๐๐ฎ๐๐ฒ๐
โช๏ธDS bench has 466 data analysis tasks and 74 data modelling tasks
โช๏ธThe tasks are sourced from ModelOff and Kaggle, the platforms hosting the most popular data science competitions
โช๏ธDifference with previous DS benchmarks:
โถ This benchmark leverages various modalities on top of text: images, Excel files, tables
โท Complex tables: sometimes several tables should be leveraged to answer one question
โธ The context is richer, with longer descriptions.
โช๏ธ Evaluation metrics : the benchmark is scored with an LLM as a judge, using a specific prompt.
๐๐ป๐๐ถ๐ด๐ต๐๐ ๐ณ๐ฟ๐ผ๐บ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ป๐ด ๐ฎ๐ด๐ฒ๐ป๐๐
โช๏ธ Their evaluation confirms that using LLMs in an agent setup, for instance by allowing them to run a single step of code execution, is more costly (especially with multi-turn frameworks like autogen) but also much more performant than the vanilla LLM.
โช๏ธ The sets of tasks solved by different models (like GPT-3.5 vs Llama-3-8B) has quite low overlap, which suggests that different models tend to try very different approches.
This new benchmark is really welcome, can't wait to try transformers agents on it! ๐ค
Read their full paper ๐ DSBench: How Far Are Data Science Agents to Becoming Data Science Experts? (2409.07703)
posted
an
update
6 days ago
Post
1039
๐๐ฉ๐๐ง๐๐ ๐๐ข๐ง๐๐ฅ๐ฅ๐ฒ ๐ซ๐๐ฏ๐๐๐ฅ๐ฌ โ๐โ: ๐๐ซ๐๐ณ๐ฒ ๐๐ก๐๐ข๐ง-๐จ๐-๐ญ๐ก๐จ๐ฎ๐ ๐ก๐ญ-๐ญ๐ฎ๐ง๐๐ ๐ฆ๐จ๐๐๐ฅ >> ๐๐๐-๐๐จ ๐ฅ
OpenAI had hinted at a mysterious โproject strawberryโ for a long time: ๐๐ต๐ฒ๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐๐ต๐ถ๐ ๐ป๐ฒ๐ ๐บ๐ผ๐ฑ๐ฒ๐น ๐ฐ๐ฎ๐น๐น๐ฒ๐ฑ โ๐ผ๐ญโ ๐ญ๐ต๐ผ๐๐ฟ ๐ฎ๐ด๐ผ, ๐ฎ๐ป๐ฑ ๐๐ต๐ฒ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐ถ๐ ๐ท๐๐๐ ๐บ๐ถ๐ป๐ฑ-๐ฏ๐น๐ผ๐๐ถ๐ป๐ด.
๐คฏ Ranks among the top 500 students in the US in a qualifier for the USA Math Olympiad
๐คฏ Beats human-PhD-level accuracy by 8% on GPQA, hard science problems benchmark where the previous best was Claude 3.5 Sonnet with 59.4.
๐คฏ Scores 78.2% on vision benchmark MMMU, making it the first model competitive w/ human experts
๐คฏ GPT-4o on MATH scored 60% โ o1 scores 95%
How did they pull this? Sadly OpenAI keeps increasing their performance in โmaking cryptic AF reports to not reveal any real infoโ, so here are excerpts:
๐ฌ โ๐ผ๐ญ ๐๐๐ฒ๐ ๐ฎ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐๐ต๐ผ๐๐ด๐ต๐ ๐๐ต๐ฒ๐ป ๐ฎ๐๐๐ฒ๐บ๐ฝ๐๐ถ๐ป๐ด ๐๐ผ ๐๐ผ๐น๐๐ฒ ๐ฎ ๐ฝ๐ฟ๐ผ๐ฏ๐น๐ฒ๐บ. ๐ง๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฟ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐น๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด, ๐ผ๐ญ ๐น๐ฒ๐ฎ๐ฟ๐ป๐ ๐๐ผ ๐ต๐ผ๐ป๐ฒ ๐ถ๐๐ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐๐ต๐ผ๐๐ด๐ต๐ ๐ฎ๐ป๐ฑ ๐ฟ๐ฒ๐ณ๐ถ๐ป๐ฒ ๐๐ต๐ฒ ๐๐๐ฟ๐ฎ๐๐ฒ๐ด๐ถ๐ฒ๐ ๐ถ๐ ๐๐๐ฒ๐. It learns to recognize and correct its mistakes.โ
And of course, they decide to hide the content of this precious Chain-of-
Thought. Would it be for maximum profit? Of course not, you awful capitalist, itโs to protect users:
๐ฌ โWe also do not want to make an unaligned chain of thought directly visible to users.โ
Theyโre right, it would certainly have hurt my feelings to see the internal of this model tearing apart math problems.
๐ค I suspect it could be not only CoT, but also some agentic behaviour where the model can just call a code executor. The kind of score improvement the show certainly looks like the ones you see with agents.
This model will be immediately released for ChatGPT and some โtrusted API usersโ.
Letโs start cooking to release the same thing in 6 months! ๐
OpenAI had hinted at a mysterious โproject strawberryโ for a long time: ๐๐ต๐ฒ๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐๐ต๐ถ๐ ๐ป๐ฒ๐ ๐บ๐ผ๐ฑ๐ฒ๐น ๐ฐ๐ฎ๐น๐น๐ฒ๐ฑ โ๐ผ๐ญโ ๐ญ๐ต๐ผ๐๐ฟ ๐ฎ๐ด๐ผ, ๐ฎ๐ป๐ฑ ๐๐ต๐ฒ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐ถ๐ ๐ท๐๐๐ ๐บ๐ถ๐ป๐ฑ-๐ฏ๐น๐ผ๐๐ถ๐ป๐ด.
๐คฏ Ranks among the top 500 students in the US in a qualifier for the USA Math Olympiad
๐คฏ Beats human-PhD-level accuracy by 8% on GPQA, hard science problems benchmark where the previous best was Claude 3.5 Sonnet with 59.4.
๐คฏ Scores 78.2% on vision benchmark MMMU, making it the first model competitive w/ human experts
๐คฏ GPT-4o on MATH scored 60% โ o1 scores 95%
How did they pull this? Sadly OpenAI keeps increasing their performance in โmaking cryptic AF reports to not reveal any real infoโ, so here are excerpts:
๐ฌ โ๐ผ๐ญ ๐๐๐ฒ๐ ๐ฎ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐๐ต๐ผ๐๐ด๐ต๐ ๐๐ต๐ฒ๐ป ๐ฎ๐๐๐ฒ๐บ๐ฝ๐๐ถ๐ป๐ด ๐๐ผ ๐๐ผ๐น๐๐ฒ ๐ฎ ๐ฝ๐ฟ๐ผ๐ฏ๐น๐ฒ๐บ. ๐ง๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฟ๐ฒ๐ถ๐ป๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐บ๐ฒ๐ป๐ ๐น๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด, ๐ผ๐ญ ๐น๐ฒ๐ฎ๐ฟ๐ป๐ ๐๐ผ ๐ต๐ผ๐ป๐ฒ ๐ถ๐๐ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ผ๐ณ ๐๐ต๐ผ๐๐ด๐ต๐ ๐ฎ๐ป๐ฑ ๐ฟ๐ฒ๐ณ๐ถ๐ป๐ฒ ๐๐ต๐ฒ ๐๐๐ฟ๐ฎ๐๐ฒ๐ด๐ถ๐ฒ๐ ๐ถ๐ ๐๐๐ฒ๐. It learns to recognize and correct its mistakes.โ
And of course, they decide to hide the content of this precious Chain-of-
Thought. Would it be for maximum profit? Of course not, you awful capitalist, itโs to protect users:
๐ฌ โWe also do not want to make an unaligned chain of thought directly visible to users.โ
Theyโre right, it would certainly have hurt my feelings to see the internal of this model tearing apart math problems.
๐ค I suspect it could be not only CoT, but also some agentic behaviour where the model can just call a code executor. The kind of score improvement the show certainly looks like the ones you see with agents.
This model will be immediately released for ChatGPT and some โtrusted API usersโ.
Letโs start cooking to release the same thing in 6 months! ๐
- 1 reply
posted
an
update
6 days ago
Post
632
๐๐
๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ป๐ด ๐๐ผ๐๐ฟ ๐๐ง๐ ๐ ๐๐ฒ๐ฏ๐ฝ๐ฎ๐ด๐ฒ๐ ๐๐ผ ๐บ๐ฎ๐ฟ๐ธ๐ฑ๐ผ๐๐ป ๐ถ๐ ๐ป๐ผ๐ ๐ฝ๐ผ๐๐๐ถ๐ฏ๐น๐ฒ ๐ฒ๐ป๐ฑ-๐๐ผ-๐ฒ๐ป๐ฑ ๐๐ถ๐๐ต ๐ฎ ๐๐ถ๐บ๐ฝ๐น๐ฒ ๐๐๐ ! ๐
Jina just released Reader-LM, that handles the whole pipeline of extracting markdown from HTML webpages.
A while ago, Jina had released a completely code-based deterministic program to do this extraction, based on some heuristics : e.g., โif the text is in a <p> tag, keep it, but if itโs hidden behind another, remove itโ.
๐ค But they received complaints from readers: some found it too detailed, other not enough, depending on the pages.
โก๏ธ So they decided, ๐บ๐ฎ๐๐ฏ๐ฒ ๐ต๐ฒ๐๐ฟ๐ถ๐๐๐ถ๐ฐ๐ ๐๐ฒ๐ฟ๐ฒ ๐ป๐ผ๐ ๐ฒ๐ป๐ผ๐๐ด๐ต: ๐ถ๐ป๐๐๐ฒ๐ฎ๐ฑ, ๐๐ต๐ฒ๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ฟ๐ฎ๐ถ๐ป ๐ฎ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ ๐ฒ๐ ๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป. This LLM does not need to be very strong,but it should handle a very long context: itโs a challenging, โshallow-but-wideโ architecture.
๐ง๐ฒ๐ฐ๐ต๐ป๐ถ๐ฐ๐ฎ๐น ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
2๏ธโฃ models: Reader-LM-0.5B and 1.5B
โ๏ธ Two stages of training: first, short and simple HTML to get the basics, then ramp up to longer and harder HTML up to 128k tokens
๐ Use contrastive search for decoding: this empirically reduces โrepeating outputโ issues
โก๏ธ Their models beat much larger models at HTML extraction ๐ฅ
๐ค Weights available on HF (sadly cc-by-nc license): jinaai/reader-lm-1.5b
Jina just released Reader-LM, that handles the whole pipeline of extracting markdown from HTML webpages.
A while ago, Jina had released a completely code-based deterministic program to do this extraction, based on some heuristics : e.g., โif the text is in a <p> tag, keep it, but if itโs hidden behind another, remove itโ.
๐ค But they received complaints from readers: some found it too detailed, other not enough, depending on the pages.
โก๏ธ So they decided, ๐บ๐ฎ๐๐ฏ๐ฒ ๐ต๐ฒ๐๐ฟ๐ถ๐๐๐ถ๐ฐ๐ ๐๐ฒ๐ฟ๐ฒ ๐ป๐ผ๐ ๐ฒ๐ป๐ผ๐๐ด๐ต: ๐ถ๐ป๐๐๐ฒ๐ฎ๐ฑ, ๐๐ต๐ฒ๐ ๐๐ฟ๐ถ๐ฒ๐ฑ ๐๐ผ ๐๐ฟ๐ฎ๐ถ๐ป ๐ฎ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ ๐ฒ๐ ๐๐ฟ๐ฎ๐ฐ๐๐ถ๐ผ๐ป. This LLM does not need to be very strong,but it should handle a very long context: itโs a challenging, โshallow-but-wideโ architecture.
๐ง๐ฒ๐ฐ๐ต๐ป๐ถ๐ฐ๐ฎ๐น ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
2๏ธโฃ models: Reader-LM-0.5B and 1.5B
โ๏ธ Two stages of training: first, short and simple HTML to get the basics, then ramp up to longer and harder HTML up to 128k tokens
๐ Use contrastive search for decoding: this empirically reduces โrepeating outputโ issues
โก๏ธ Their models beat much larger models at HTML extraction ๐ฅ
๐ค Weights available on HF (sadly cc-by-nc license): jinaai/reader-lm-1.5b
- 1 reply
posted
an
update
7 days ago
Post
1297
๐ข๐ฝ๐ฒ๐ป ๐๐๐ ๐ ๐ฎ๐ฟ๐ฒ ๐ผ๐ป ๐ณ๐ถ๐ฟ๐ฒ ๐ฟ๐ถ๐ด๐ต๐ ๐ป๐ผ๐! ๐ฅ ๐๐ฒ๐ฒ๐ฝ๐ฆ๐ฒ๐ฒ๐ธ-๐ฉ๐ฎ.๐ฑ ๐ฎ๐ป๐ฑ ๐ผ๐๐ต๐ฒ๐ฟ ๐๐ผ๐ฝ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ๐
Mistral AI just released Pixtral-12B, a vision models that seems to perform extremely well! From Mistralโs own benchmark, it beats the great Qwen2-7B and Llava-OV.
๐ค But Mistralโs benchmarks evaluate in Chain-of-Thought, and even in CoT they show lower scores for other models than the scores already published in non-CoT, which is very strangeโฆ Evaluation is not a settled science!
But itโs only the last of a flurry of great models. Here are the ones currently squatting the top of the Models Hub page:
โถ ๐ ๐๐ฅ๐๐ฆ๐-๐.๐-๐๐ ๐๐ฆ๐ง๐ข, a model built upon Llama-3.1-8B-Instruct, that simultaneously generates text and speech response with an extremely low latency of 250ms (Moshi, Kyutaiโs 8B, did 140ms)
โท ๐๐ฃ๏ธ ๐ ๐ข๐ฌ๐ก ๐๐ฉ๐๐๐๐ก ๐ฏ๐.๐, text-to-speech model that supports 8 languages ๐ฌ๐ง๐จ๐ณ๐ฉ๐ช๐ฏ๐ต๐ซ๐ท๐ช๐ธ๐ฐ๐ท๐ธ๐ฆ with extremely good quality for a light size (~1GB weights) and low latency
โธ ๐ณ ๐๐๐๐ฉ๐๐๐๐ค-๐๐.๐, a 236B model with 128k context length that combines the best of DeepSeek-V2-Chat and the more recent DeepSeek-Coder-V2-Instruct. Depending on benchmarks, it ranks just below Llama-3.1-405B. Released with custom โdeepseekโ license, quite commercially permissive.
โน ๐๐จ๐ฅ๐๐ซ ๐๐ซ๐จ published by Upstage: a 22B model (so inference fits on a single GPU) that comes just under Llama-3.1-70B performance : MMLU: 79, GPQA: 36, IFEval: 84
โบ ๐๐ข๐ง๐ข๐๐๐๐-๐๐, a small model that claims very impressive scores, even beating much larger models like Llama-3.1-8B. Let's wait for more scores because these look too good!
Letโs keep looking, more good stuff is coming our way ๐ญ
Mistral AI just released Pixtral-12B, a vision models that seems to perform extremely well! From Mistralโs own benchmark, it beats the great Qwen2-7B and Llava-OV.
๐ค But Mistralโs benchmarks evaluate in Chain-of-Thought, and even in CoT they show lower scores for other models than the scores already published in non-CoT, which is very strangeโฆ Evaluation is not a settled science!
But itโs only the last of a flurry of great models. Here are the ones currently squatting the top of the Models Hub page:
โถ ๐ ๐๐ฅ๐๐ฆ๐-๐.๐-๐๐ ๐๐ฆ๐ง๐ข, a model built upon Llama-3.1-8B-Instruct, that simultaneously generates text and speech response with an extremely low latency of 250ms (Moshi, Kyutaiโs 8B, did 140ms)
โท ๐๐ฃ๏ธ ๐ ๐ข๐ฌ๐ก ๐๐ฉ๐๐๐๐ก ๐ฏ๐.๐, text-to-speech model that supports 8 languages ๐ฌ๐ง๐จ๐ณ๐ฉ๐ช๐ฏ๐ต๐ซ๐ท๐ช๐ธ๐ฐ๐ท๐ธ๐ฆ with extremely good quality for a light size (~1GB weights) and low latency
โธ ๐ณ ๐๐๐๐ฉ๐๐๐๐ค-๐๐.๐, a 236B model with 128k context length that combines the best of DeepSeek-V2-Chat and the more recent DeepSeek-Coder-V2-Instruct. Depending on benchmarks, it ranks just below Llama-3.1-405B. Released with custom โdeepseekโ license, quite commercially permissive.
โน ๐๐จ๐ฅ๐๐ซ ๐๐ซ๐จ published by Upstage: a 22B model (so inference fits on a single GPU) that comes just under Llama-3.1-70B performance : MMLU: 79, GPQA: 36, IFEval: 84
โบ ๐๐ข๐ง๐ข๐๐๐๐-๐๐, a small model that claims very impressive scores, even beating much larger models like Llama-3.1-8B. Let's wait for more scores because these look too good!
Letโs keep looking, more good stuff is coming our way ๐ญ
posted
an
update
7 days ago
Post
2137
๐๐ฟ๐ฐ๐ฒ๐ฒ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ๐ ๐ฆ๐๐ฝ๐ฒ๐ฟ๐ก๐ผ๐๐ฎ, ๐ฏ๐ฒ๐๐๐ฒ๐ฟ ๐ณ๐ถ๐ป๐ฒ-๐๐๐ป๐ฒ ๐ผ๐ณ ๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ญ-๐ณ๐ฌ๐!
2๏ธโฃ versions: 70B and 8B
๐ง Trained by distilling logits from Llama-3.1-405B
๐ฅ Used a clever compression method to reduce dataset weight from 2.9 Petabytes down to 50GB (may share it in a paper)
โ๏ธ Not all benchmarks are improved: GPQA and MUSR go down a slight bit
๐ค 8B weights are available on HF (not the 70B)
Read their blog post ๐ https://blog.arcee.ai/arcee-supernova-training-pipeline-and-model-composition/
Model weights (8B) ๐ arcee-ai/Llama-3.1-SuperNova-Lite
2๏ธโฃ versions: 70B and 8B
๐ง Trained by distilling logits from Llama-3.1-405B
๐ฅ Used a clever compression method to reduce dataset weight from 2.9 Petabytes down to 50GB (may share it in a paper)
โ๏ธ Not all benchmarks are improved: GPQA and MUSR go down a slight bit
๐ค 8B weights are available on HF (not the 70B)
Read their blog post ๐ https://blog.arcee.ai/arcee-supernova-training-pipeline-and-model-composition/
Model weights (8B) ๐ arcee-ai/Llama-3.1-SuperNova-Lite
posted
an
update
8 days ago
Post
607
> ๐ช๐ฎ๐ป๐ ๐๐ผ ๐ธ๐ป๐ผ๐ ๐ต๐ผ๐ ๐บ๐๐ฐ๐ต ๐ฎ๐ป ๐๐ฃ๐ ๐๐๐ ๐ฐ๐ฎ๐น๐น ๐ฐ๐ผ๐๐๐ ๐๐ผ๐?
I've just made this Space that gets you the API price for any LLM call, for nearly all inference providers out there!
This is based on a comment by @victor under my HF Post a few months back, and leverages BerriAI's data for LLM prices.
Check it out here ๐ m-ric/text_to_dollars
I've just made this Space that gets you the API price for any LLM call, for nearly all inference providers out there!
This is based on a comment by @victor under my HF Post a few months back, and leverages BerriAI's data for LLM prices.
Check it out here ๐ m-ric/text_to_dollars
replied to
their
post
8 days ago
FYI, I've finally built this here: https://huggingface.co/spaces/m-ric/text_to_dollars
replied to
their
post
9 days ago
Read the article here: https://www.adyen.com/knowledge-hub/llm-inference-at-scale-with-tgi
posted
an
update
9 days ago
Post
1158
> Article read: Simple guide to LLM inference and to TGI
I've just read article "LLM inference at scale with TGI" by @martinigoyanes . It's really good content, a must-read if you want a good low-level intro to LLM inference with TGI!
My takeaways:
How does inference work?
๐ง Prefill: the input prompt is tokenized on CPU, then transferred to GPU. Then one single forward pass generates the initial token.
๐ Decode: the model generates ("decodes") tokens one by one, each time appending the new token to the current input of size N to then generate a new token again with this augmented input of length N+1. This loop ends either when a specific token called "End-of-sequence" is generated or when the completion reaches a pre-specified maximum length. Then the sequence is de-tokenized on CPU to yield text again.
โฑ๏ธ This step's speed determines the Time Per Output Token, which directly translates to the key metric: Throughput
๐ค How was the separation between the two steps decided ? Like, why does prefill include this strange generation of only one token at then end?
โก๏ธ The cost of attention scales quadratically with the number of tokens, so it can really explode quickly.
To compensate for that, a really important technique called KV caching was devised: using the fact that when generating token N+1, the Key and Value (K and V) matrices generated inside the Transformers are a simple extension from the K and V from the previous step, the model caches the K and V matrices between steps : thus the separation - the prefill part is the part that prepares this KV cache, while the decoding is the one that leverages it and expands it by one at each step.
TGI-specific takeaways:
โ๏ธ TGI has many SOTA techniques for decoding: Paged Attention, KV Caching and Flash Attentionโฆ
๐ TGI's router handles generations finishing early because of an EOS token: instead of static batching, it continuously batches requests to the inference engine & filters away finished requests.
I've just read article "LLM inference at scale with TGI" by @martinigoyanes . It's really good content, a must-read if you want a good low-level intro to LLM inference with TGI!
My takeaways:
How does inference work?
๐ง Prefill: the input prompt is tokenized on CPU, then transferred to GPU. Then one single forward pass generates the initial token.
๐ Decode: the model generates ("decodes") tokens one by one, each time appending the new token to the current input of size N to then generate a new token again with this augmented input of length N+1. This loop ends either when a specific token called "End-of-sequence" is generated or when the completion reaches a pre-specified maximum length. Then the sequence is de-tokenized on CPU to yield text again.
โฑ๏ธ This step's speed determines the Time Per Output Token, which directly translates to the key metric: Throughput
๐ค How was the separation between the two steps decided ? Like, why does prefill include this strange generation of only one token at then end?
โก๏ธ The cost of attention scales quadratically with the number of tokens, so it can really explode quickly.
To compensate for that, a really important technique called KV caching was devised: using the fact that when generating token N+1, the Key and Value (K and V) matrices generated inside the Transformers are a simple extension from the K and V from the previous step, the model caches the K and V matrices between steps : thus the separation - the prefill part is the part that prepares this KV cache, while the decoding is the one that leverages it and expands it by one at each step.
TGI-specific takeaways:
โ๏ธ TGI has many SOTA techniques for decoding: Paged Attention, KV Caching and Flash Attentionโฆ
๐ TGI's router handles generations finishing early because of an EOS token: instead of static batching, it continuously batches requests to the inference engine & filters away finished requests.
-
- 1 reply
posted
an
update
13 days ago
Post
1883
๐คฏ ๐ ๐ป๐ฒ๐ ๐ณ๐ฌ๐ ๐ผ๐ฝ๐ฒ๐ป-๐๐ฒ๐ถ๐ด๐ต๐๐ ๐๐๐ ๐ฏ๐ฒ๐ฎ๐๐ ๐๐น๐ฎ๐๐ฑ๐ฒ-๐ฏ.๐ฑ-๐ฆ๐ผ๐ป๐ป๐ฒ๐ ๐ฎ๐ป๐ฑ ๐๐ฃ๐ง-๐ฐ๐ผ!
@mattshumer , CEO from Hyperwrite AI, had an idea he wanted to try out: why not fine-tune LLMs to always output their thoughts in specific parts, delineated by <thinking> tags?
Even better: inside of that, you could nest other sections, to reflect critically on previous output. Letโs name this part <reflection>. Planning is also put in a separate step.
He named the method โReflection tuningโ and set out to fine-tune a Llama-3.1-70B with it.
Well it turns out, it works mind-boggingly well!
๐คฏ Reflection-70B beats GPT-4o, Sonnet-3.5, and even the much bigger Llama-3.1-405B!
๐ง๐;๐๐ฅ
๐ฅ This new 70B open-weights model beats GPT-4o, Claude Sonnet, et al.
โฐ 405B in training, coming soon
๐ Report coming next week
โ๏ธ Uses GlaiveAI synthetic data
๐ค Available on HF!
Iโm starting an Inference Endpoint right now for this model to give it a spin!
Check it out ๐ mattshumer/Reflection-Llama-3.1-70B
@mattshumer , CEO from Hyperwrite AI, had an idea he wanted to try out: why not fine-tune LLMs to always output their thoughts in specific parts, delineated by <thinking> tags?
Even better: inside of that, you could nest other sections, to reflect critically on previous output. Letโs name this part <reflection>. Planning is also put in a separate step.
He named the method โReflection tuningโ and set out to fine-tune a Llama-3.1-70B with it.
Well it turns out, it works mind-boggingly well!
๐คฏ Reflection-70B beats GPT-4o, Sonnet-3.5, and even the much bigger Llama-3.1-405B!
๐ง๐;๐๐ฅ
๐ฅ This new 70B open-weights model beats GPT-4o, Claude Sonnet, et al.
โฐ 405B in training, coming soon
๐ Report coming next week
โ๏ธ Uses GlaiveAI synthetic data
๐ค Available on HF!
Iโm starting an Inference Endpoint right now for this model to give it a spin!
Check it out ๐ mattshumer/Reflection-Llama-3.1-70B
- 3 replies
posted
an
update
13 days ago
Post
819
๐ย ๐ช๐ต๐ฒ๐ฟ๐ฒ ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐น๐ฎ๐๐ ๐ฎ๐ฟ๐ฒ ๐๐ฎ๐ธ๐ถ๐ป๐ด ๐๐ : ๐ฏ๐ ๐ฎ๐ฌ๐ฎ๐ด, ๐๐ ๐๐น๐๐๐๐ฒ๐ฟ๐ ๐๐ถ๐น๐น ๐ฟ๐ฒ๐ฎ๐ฐ๐ต ๐๐ต๐ฒ ๐ฝ๐ผ๐๐ฒ๐ฟ ๐ฐ๐ผ๐ป๐๐๐บ๐ฝ๐๐ถ๐ผ๐ป ๐ผ๐ณ ๐ฒ๐ป๐๐ถ๐ฟ๐ฒ ๐ฐ๐ผ๐๐ป๐๐ฟ๐ถ๐ฒ๐
Reminder : โScaling lawsโ are empirical laws saying that if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
๐๏ธย As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So letโs take this progress a bit further:
- 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
- 2024: today, companies start training on much larger clusters like the โsuper AI clusterโ of Elon Muskโs xAI, 10^27 FLOPS, $1B
- 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
- 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries?
โก๏ธย Three key ingredients of training might be a roadblock to scaling up :
๐ธย Money: but itโs very unlikely, given the potential market size for AGI, that investors lose interest.
โก๏ธ Energy supply at a specific location
๐ย Training data: weโre already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
๐คย Iโd be curious to hear your thoughts: do you think weโll race all the way there?
Reminder : โScaling lawsโ are empirical laws saying that if you keep multiplying your compute by x10, your models will mechanically keep getting better and better.
To give you an idea, GPT-3 can barely write sentences, and GPT-4, which only used x15 its amount of compute, already sounds much smarter than some of my friends (although it's not really - or at least I haven't tested them side-by side). So you can imagine how far a x100 over GPT-4 can take us.
๐๏ธย As a result, tech titans are racing to build the biggest models, and for this they need gigantic training clusters.
The picture below shows the growth of training compute: it is increasing at a steady exponential rate of a x10 every 2 years. So letโs take this progress a bit further:
- 2022: starting training for GPT-4 : 10^26 FLOPs, cost of $100M
- 2024: today, companies start training on much larger clusters like the โsuper AI clusterโ of Elon Muskโs xAI, 10^27 FLOPS, $1B
- 2026 : by then clusters will require 1GW, i.e. around the full power generated by a nuclear reactor
- 2028: we reach cluster prices in the 100 billion dollars, using 10GW, more than the most powerful power stations currently in use in the US. This last size seems crazy, but Microsoft and OpenAI already are planning one.
Will AI clusters effectively reach these crazy sizes where the consume as much as entire countries?
โก๏ธย Three key ingredients of training might be a roadblock to scaling up :
๐ธย Money: but itโs very unlikely, given the potential market size for AGI, that investors lose interest.
โก๏ธ Energy supply at a specific location
๐ย Training data: weโre already using 15 trillion tokens for Llama-3.1 when Internet has something like 60 trillion.
๐คย Iโd be curious to hear your thoughts: do you think weโll race all the way there?
- 3 replies
posted
an
update
14 days ago
Post
2093
๐ฅณ ๐ง๐ฟ๐ฎ๐ป๐๐ณ๐ผ๐ฟ๐บ๐ฒ๐ฟ๐ ๐๐ด๐ฒ๐ป๐๐ ๐ป๐ผ๐ ๐๐๐ฝ๐ฝ๐ผ๐ฟ๐๐ ๐ ๐๐น๐๐ถ-๐ฎ๐ด๐ฒ๐ป๐ ๐๐๐๐๐ฒ๐บ๐!
Multi-agent systems have been introduced in Microsoft's framework Autogen. It simply means having several agents working together to solve your task instead of only one : this paradigm empirically yields better performance on most benchmarks. The reason for this better performance is conceptually simple: for many tasks, rather than using a do-it-all system, you would prefer to specialize units on sub-tasks. Here, having agents with separate tool sets and memories allows to achieve efficient specialization.
You can now easily build hierarchical multi-agent systems with transformers.agents (not released yet, use the dev version)
To do so, encapsulate the agent in a ManagedAgent object. This object needs arguments agent, name, and a description, which will then be embedded in the manager agent's system prompt to let it know how to call this managed agent, as we also do for tools.
Cf the example in the image! We'll keep building on this paradigm in the upcoming weeks ๐
Read more in the doc ๐ https://github.com/huggingface/transformers/blob/main/docs/source/en/agents_advanced.md
Checkout an advanced multi-agent system that tops the GAIA leaderboard ๐ https://github.com/aymeric-roucher/GAIA/blob/main/gaia_multiagent.py
Multi-agent systems have been introduced in Microsoft's framework Autogen. It simply means having several agents working together to solve your task instead of only one : this paradigm empirically yields better performance on most benchmarks. The reason for this better performance is conceptually simple: for many tasks, rather than using a do-it-all system, you would prefer to specialize units on sub-tasks. Here, having agents with separate tool sets and memories allows to achieve efficient specialization.
You can now easily build hierarchical multi-agent systems with transformers.agents (not released yet, use the dev version)
To do so, encapsulate the agent in a ManagedAgent object. This object needs arguments agent, name, and a description, which will then be embedded in the manager agent's system prompt to let it know how to call this managed agent, as we also do for tools.
Cf the example in the image! We'll keep building on this paradigm in the upcoming weeks ๐
Read more in the doc ๐ https://github.com/huggingface/transformers/blob/main/docs/source/en/agents_advanced.md
Checkout an advanced multi-agent system that tops the GAIA leaderboard ๐ https://github.com/aymeric-roucher/GAIA/blob/main/gaia_multiagent.py
posted
an
update
15 days ago
Post
792
๐จ ๐๐๐บ๐ฎ๐ป ๐๐ฒ๐ฒ๐ฑ๐ฏ๐ฎ๐ฐ๐ธ ๐ณ๐ผ๐ฟ ๐๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด: ๐ก๐ผ๐ ๐๐ต๐ฒ ๐ด๐ผ๐น๐ฑ๐ฒ๐ป ๐ด๐ผ๐ผ๐๐ฒ ๐๐ฒ ๐๐ต๐ผ๐๐ด๐ต๐?
Iโve just read a great paper where Cohere researchers raises significant questions about using Human feedback to evaluate AI language models.
Human feedback is often regarded as the gold standard for judging AI performance, but it turns out, it might be more like fool's gold : the study reveals that our human judgments are easily swayed by factors that have nothing to do with actual AI performance.
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ง Test several models: Llama-2, Falcon-40B, Cohere Command 6 and 52B ๐ โโ๏ธ Refusing to answer tanks AI ratings more than getting facts wrong. We apparently prefer a wrong answer to no answer!
๐ช Confidence is key (even when it shouldn't be): More assertive AI responses are seen as more factual, even when they're not. This could be pushing AI development in the wrong direction, with systems like RLHF.
๐ญ The assertiveness trap: As AI responses get more confident-sounding, non-expert annotators become less likely to notice when they're wrong or inconsistent.
And a consequence of the above:
๐ ๐ฅ๐๐๐ ๐บ๐ถ๐ด๐ต๐ ๐ฏ๐ฎ๐ฐ๐ธ๐ณ๐ถ๐ฟ๐ฒ: Using human feedback to train AI (Reinforcement Learning from Human Feedback) could accidentally make AI more overconfident and less accurate.
This paper means we need to think carefully about how we evaluate and train AI systems to ensure we're rewarding correctness over apparences of it like confident talk.
โ๏ธ Chatbot Arenaโs ELO leaderboard, based on crowdsourced answers from average joes like you and me, might become completely irrelevant as models will become smarter and smarter.
Read the paper ๐ Human Feedback is not Gold Standard (2309.16349)
Iโve just read a great paper where Cohere researchers raises significant questions about using Human feedback to evaluate AI language models.
Human feedback is often regarded as the gold standard for judging AI performance, but it turns out, it might be more like fool's gold : the study reveals that our human judgments are easily swayed by factors that have nothing to do with actual AI performance.
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ง Test several models: Llama-2, Falcon-40B, Cohere Command 6 and 52B ๐ โโ๏ธ Refusing to answer tanks AI ratings more than getting facts wrong. We apparently prefer a wrong answer to no answer!
๐ช Confidence is key (even when it shouldn't be): More assertive AI responses are seen as more factual, even when they're not. This could be pushing AI development in the wrong direction, with systems like RLHF.
๐ญ The assertiveness trap: As AI responses get more confident-sounding, non-expert annotators become less likely to notice when they're wrong or inconsistent.
And a consequence of the above:
๐ ๐ฅ๐๐๐ ๐บ๐ถ๐ด๐ต๐ ๐ฏ๐ฎ๐ฐ๐ธ๐ณ๐ถ๐ฟ๐ฒ: Using human feedback to train AI (Reinforcement Learning from Human Feedback) could accidentally make AI more overconfident and less accurate.
This paper means we need to think carefully about how we evaluate and train AI systems to ensure we're rewarding correctness over apparences of it like confident talk.
โ๏ธ Chatbot Arenaโs ELO leaderboard, based on crowdsourced answers from average joes like you and me, might become completely irrelevant as models will become smarter and smarter.
Read the paper ๐ Human Feedback is not Gold Standard (2309.16349)
posted
an
update
16 days ago
Post
2192
๐ค ๐ง๐ต๐ฒ ๐๐ ๐ฆ๐ฐ๐ถ๐ฒ๐ป๐๐ถ๐๐: ๐๐ด๐ฒ๐ป๐๐ถ๐ฐ, ๐ณ๐๐น๐น๐-๐ฎ๐๐๐ผ๐บ๐ฎ๐๐ฒ๐ฑ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฝ๐ถ๐ฝ๐ฒ๐น๐ถ๐ป๐ฒ ๐ณ๐ผ๐ฟ ๐๐ป๐ฑ๐ฒ๐ฟ $๐ญ๐ฑ ๐ฝ๐ฒ๐ฟ ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ
Researchers have just created an AI system that ๐ฐ๐ฎ๐ป ๐ฐ๐ผ๐ป๐ฑ๐๐ฐ๐ ๐ฒ๐ป๐๐ถ๐ฟ๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฝ๐ฟ๐ผ๐ท๐ฒ๐ฐ๐๐ ๐ณ๐ฟ๐ผ๐บ ๐๐๐ฎ๐ฟ๐ ๐๐ผ ๐ณ๐ถ๐ป๐ถ๐๐ต, ๐ฝ๐ผ๐๐ฒ๐ป๐๐ถ๐ฎ๐น๐น๐ ๐ฟ๐ฒ๐๐ผ๐น๐๐๐ถ๐ผ๐ป๐ถ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ฐ๐ถ๐ฒ๐ป๐๐ถ๐ณ๐ถ๐ฐ ๐ฑ๐ถ๐๐ฐ๐ผ๐๐ฒ๐ฟ๐ถ๐ฒ๐ ๐ฎ๐ฟ๐ฒ ๐บ๐ฎ๐ฑ๐ฒ.
It doesn't just assist with specific tasks - it automates the entire research process, from generating ideas to writing and reviewing papers.
1 - brainstorm novel research directions, 2- write and execute code for experiments & visualize results, get references, and even 3- write up findings in a full academic paper format!
And it can do all this for under $15 per paper! ๐คฏ
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ง Generates novel research ideas across multiple topics (e.g. diffusion modeling, transformers, learning dynamics aka โgrokkingโ)
๐จโ๐ป Uses open-source coding assistant Aider to implement ideas and run experiments. This is especially important since this agentic assistant can iterate if it fails somewhere.
๐ Visualizes results and plans follow-up experiments (up to 5 rounds)
โ๏ธ Writes full academic papers, including finding references using Semantic Search API
๐ต๏ธ Runs a simulated peer review process to evaluate paper quality
๐ฐ Total cost per paper is under $15. This system can generate "hundreds of interesting, medium-quality papers" in just a week !
๐ฆ๐๐ถ๐น๐น ๐ป๐ผ๐ ๐ฟ๐ฒ๐ฎ๐ฑ๐ ๐๐ผ ๐ณ๐ถ๐น๐น ๐๐๐๐ฅ ๐๐ถ๐๐ต ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ๐:
๐ Ideas generated in one domain tend to be repetitive across different runs, and even different language model
๐ Does not use vision capabilities to fix visual issues in plots
๐ญ Models occasionally hallucinate entire results tables
โ Only few of the generated papers would actually meet the threshold for acceptance at a top AI conference
๐ย Read their paper: The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (2408.06292)
Researchers have just created an AI system that ๐ฐ๐ฎ๐ป ๐ฐ๐ผ๐ป๐ฑ๐๐ฐ๐ ๐ฒ๐ป๐๐ถ๐ฟ๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฝ๐ฟ๐ผ๐ท๐ฒ๐ฐ๐๐ ๐ณ๐ฟ๐ผ๐บ ๐๐๐ฎ๐ฟ๐ ๐๐ผ ๐ณ๐ถ๐ป๐ถ๐๐ต, ๐ฝ๐ผ๐๐ฒ๐ป๐๐ถ๐ฎ๐น๐น๐ ๐ฟ๐ฒ๐๐ผ๐น๐๐๐ถ๐ผ๐ป๐ถ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ฐ๐ถ๐ฒ๐ป๐๐ถ๐ณ๐ถ๐ฐ ๐ฑ๐ถ๐๐ฐ๐ผ๐๐ฒ๐ฟ๐ถ๐ฒ๐ ๐ฎ๐ฟ๐ฒ ๐บ๐ฎ๐ฑ๐ฒ.
It doesn't just assist with specific tasks - it automates the entire research process, from generating ideas to writing and reviewing papers.
1 - brainstorm novel research directions, 2- write and execute code for experiments & visualize results, get references, and even 3- write up findings in a full academic paper format!
And it can do all this for under $15 per paper! ๐คฏ
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ง Generates novel research ideas across multiple topics (e.g. diffusion modeling, transformers, learning dynamics aka โgrokkingโ)
๐จโ๐ป Uses open-source coding assistant Aider to implement ideas and run experiments. This is especially important since this agentic assistant can iterate if it fails somewhere.
๐ Visualizes results and plans follow-up experiments (up to 5 rounds)
โ๏ธ Writes full academic papers, including finding references using Semantic Search API
๐ต๏ธ Runs a simulated peer review process to evaluate paper quality
๐ฐ Total cost per paper is under $15. This system can generate "hundreds of interesting, medium-quality papers" in just a week !
๐ฆ๐๐ถ๐น๐น ๐ป๐ผ๐ ๐ฟ๐ฒ๐ฎ๐ฑ๐ ๐๐ผ ๐ณ๐ถ๐น๐น ๐๐๐๐ฅ ๐๐ถ๐๐ต ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ๐:
๐ Ideas generated in one domain tend to be repetitive across different runs, and even different language model
๐ Does not use vision capabilities to fix visual issues in plots
๐ญ Models occasionally hallucinate entire results tables
โ Only few of the generated papers would actually meet the threshold for acceptance at a top AI conference
๐ย Read their paper: The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (2408.06292)
posted
an
update
22 days ago
Post
2090
๐ฎ ๐ ๐ป๐ฒ๐๐ฟ๐ฎ๐น ๐ป๐ฒ๐๐๐ผ๐ฟ๐ธ ๐๐ถ๐บ๐๐น๐ฎ๐๐ฒ๐ ๐๐ข๐ข๐ : ๐๐ผ๐ผ๐ด๐น๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต๐ฒ๐ฟ๐ ๐ผ๐ฝ๐ฒ๐ป ๐๐ต๐ฒ ๐๐ฎ๐ ๐ณ๐ผ๐ฟ ๐ฐ๐ผ๐บ๐ฝ๐น๐ฒ๐๐ฒ๐น๐-๐๐-๐ด๐ฒ๐ป๐ฒ๐ฟ๐ฎ๐๐ฒ๐ฑ ๐ด๐ฎ๐บ๐ฒ๐!
Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The playerโs in-game actions would have a real, lasting impact on the game story.
In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.
โก๏ธย They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!
Here's how they did it:
1. They trained an RL agent to play DOOM and recorded its gameplay sessions.
2. They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
3. During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ฎ๐ค Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
๐ง The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
๐ They use "noise augmentation" during training to prevent quality degradation in long play sessions
๐ The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)
The researchers did not open source the code, but I feel like weโve just seen a part of the future being written!
Their paper (exploding the upvote counter) ๐ย Diffusion Models Are Real-Time Game Engines (2408.14837)
In a similar vein, play @Jofthomas 's 'Everchanging Quest' ๐ฎ Jofthomas/Everchanging-Quest
Imagine if games were completely live-generated by an AI model : the NPCs and their dialogues, the storyline, and even the game environment. The playerโs in-game actions would have a real, lasting impact on the game story.
In a very exciting paper, Google researchers just gave us the first credible glimpse of this future.
โก๏ธย They created GameNGen, the first neural model that can simulate a complex 3D game in real-time. They use it to simulate the classic game DOOM running at over 20 frames per second on a single TPU, with image quality comparable to lossy JPEG compression. And it feels just like the true game!
Here's how they did it:
1. They trained an RL agent to play DOOM and recorded its gameplay sessions.
2. They then used these recordings to train a diffusion model to predict the next frame, based on past frames and player actions.
3. During inference, they use only 4 denoising steps (instead of the usual dozens) to generate each frame quickly.
๐๐ฒ๐ ๐ถ๐ป๐๐ถ๐ด๐ต๐๐:
๐ฎ๐ค Human players can barely tell the difference between short clips (3 seconds) of the real game or the simulation
๐ง The model maintains game state (health, ammo, etc.) over long periods despite having only 3 seconds of effective context length
๐ They use "noise augmentation" during training to prevent quality degradation in long play sessions
๐ The game runs on one TPU at 20 FPS with 4 denoising steps, or 50 FPS with model distillation (with some quality loss)
The researchers did not open source the code, but I feel like weโve just seen a part of the future being written!
Their paper (exploding the upvote counter) ๐ย Diffusion Models Are Real-Time Game Engines (2408.14837)
In a similar vein, play @Jofthomas 's 'Everchanging Quest' ๐ฎ Jofthomas/Everchanging-Quest
replied to
their
post
27 days ago
Yes that's why I added the big if behind!
posted
an
update
27 days ago
Post
906
๐๐๐ฎ๐ญ ๐ถ๐๐ฒ๐ฟ๐ฎ๐๐ฒ๐ ๐๐ถ๐๐ต ๐ป๐ฒ๐ ๐๐ฎ๐บ๐ฏ๐ฎ ๐ญ.๐ฑ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ: ๐ก๐ฒ๐ ๐๐๐ฎ๐ป๐ฑ๐ฎ๐ฟ๐ฑ ๐ณ๐ผ๐ฟ ๐น๐ผ๐ป๐ด-๐ฐ๐ผ๐ป๐๐ฒ๐
๐ ๐๐๐ฒ-๐ฐ๐ฎ๐๐ฒ๐!๐
@ai21labs used a different architecture to beat the status-quo Transformers models: Jamba architecture combines classic Transformers layers with the new Mamba layers, for which the complexity is a linear (instead of quadratic) function of the context length.
What does this imply?
โก๏ธ Jamba models are much more efficient for long contexts: faster (up to 2.5x faster for long context), takes less memory, and also performs better to recall everything in the prompt.
That means itโs a new go-to model for RAG or agentic applications!
And the performance is not too shabby: Jamba 1.5 models are comparable in perf to similar-sized Llama-3.1 models! The largest model even outperforms Llama-3.1 405B on Arena-Hard.
โ๏ธ Comes in 2 sizes: Mini (12B active/52B) and Large (94B active/399B)
๐ Both deliver 256k context length, for low memory: Jamba-1.5 mini fits 140k context length on one single A100.
โ๏ธ New quanttization method: Experts Int8 quantizes only the weights parts of the MoE layers, which account for 85% of weights
๐ค Natively supports JSON format generation & function calling.
๐ Permissive license *if your org makes <$50M revenue*
Available on the Hub ๐ ai21labs/jamba-15-66c44befa474a917fcf55251
Read their release blog post ๐ https://www.ai21.com/blog/announcing-jamba-model-family
@ai21labs used a different architecture to beat the status-quo Transformers models: Jamba architecture combines classic Transformers layers with the new Mamba layers, for which the complexity is a linear (instead of quadratic) function of the context length.
What does this imply?
โก๏ธ Jamba models are much more efficient for long contexts: faster (up to 2.5x faster for long context), takes less memory, and also performs better to recall everything in the prompt.
That means itโs a new go-to model for RAG or agentic applications!
And the performance is not too shabby: Jamba 1.5 models are comparable in perf to similar-sized Llama-3.1 models! The largest model even outperforms Llama-3.1 405B on Arena-Hard.
โ๏ธ Comes in 2 sizes: Mini (12B active/52B) and Large (94B active/399B)
๐ Both deliver 256k context length, for low memory: Jamba-1.5 mini fits 140k context length on one single A100.
โ๏ธ New quanttization method: Experts Int8 quantizes only the weights parts of the MoE layers, which account for 85% of weights
๐ค Natively supports JSON format generation & function calling.
๐ Permissive license *if your org makes <$50M revenue*
Available on the Hub ๐ ai21labs/jamba-15-66c44befa474a917fcf55251
Read their release blog post ๐ https://www.ai21.com/blog/announcing-jamba-model-family
-
- 2 replies
posted
an
update
29 days ago
Post
3359
๐๐ผ๐ผ๐ด๐น๐ฒ ๐ฝ๐ฎ๐ฝ๐ฒ๐ฟ : ๐๐ฐ๐ฎ๐น๐ถ๐ป๐ด ๐๐ฝ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐ฐ๐ผ๐บ๐ฝ๐๐๐ฒ ๐ฏ๐ฒ๐ฎ๐๐ ๐ญ๐ฐ๐
๐น๐ฎ๐ฟ๐ด๐ฒ๐ฟ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐
Remember scaling laws? These are empirical laws that say "the bigger your model, the better it gets". More precisely, "as your compute increases exponentially, loss decreases in a linear fashion". They have wild implications, suggesting that spending 100x more training compute would make you super-LLMs. That's why companies are racing to build the biggest AI superclusters ever, and Meta bought 350k H100 GPUs, which probably cost in the order of $1B.
But think of this : we're building huge reasoning machines, but only ask them to do one pass through the model to get one token of the final answer : i.e., we expend a minimal effort on inference. That's like building a Caterpillar truck and making it run on a lawnmower's motor. ๐๐ต Couldn't we optimize this? ๐ค
๐ก So instead of scaling up on training by training even bigger models on many more trillions of tokens, Google researchers explored this under-explored avenue : scaling up inference compute.
They combine two methods to use more compute : either a reviser that iterated to adapt the model distribution, or generate N different completions (for instance through Beam Search) and select only the best one using an additional verifier model.
They use a Palm-2 model (released in May 23) on the MATH dataset : Palm-2 has the advantage of getting a low performance on MATH, but not zero, so that improvements will be noticeable.
And the results show that for the same fixed amount of inference compute:
๐ฅ a smaller model with more effort on decoding beats a x14 bigger model using naive greedy sampling.
That means that you can divide your training costs by 14 and still get the same perf for the same inference cost!
Take that, scaling laws. Mark Zuckerberg, you're welcome, hope I can get some of these H100s.
Read the paper here ๐ Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2408.03314)
Remember scaling laws? These are empirical laws that say "the bigger your model, the better it gets". More precisely, "as your compute increases exponentially, loss decreases in a linear fashion". They have wild implications, suggesting that spending 100x more training compute would make you super-LLMs. That's why companies are racing to build the biggest AI superclusters ever, and Meta bought 350k H100 GPUs, which probably cost in the order of $1B.
But think of this : we're building huge reasoning machines, but only ask them to do one pass through the model to get one token of the final answer : i.e., we expend a minimal effort on inference. That's like building a Caterpillar truck and making it run on a lawnmower's motor. ๐๐ต Couldn't we optimize this? ๐ค
๐ก So instead of scaling up on training by training even bigger models on many more trillions of tokens, Google researchers explored this under-explored avenue : scaling up inference compute.
They combine two methods to use more compute : either a reviser that iterated to adapt the model distribution, or generate N different completions (for instance through Beam Search) and select only the best one using an additional verifier model.
They use a Palm-2 model (released in May 23) on the MATH dataset : Palm-2 has the advantage of getting a low performance on MATH, but not zero, so that improvements will be noticeable.
And the results show that for the same fixed amount of inference compute:
๐ฅ a smaller model with more effort on decoding beats a x14 bigger model using naive greedy sampling.
That means that you can divide your training costs by 14 and still get the same perf for the same inference cost!
Take that, scaling laws. Mark Zuckerberg, you're welcome, hope I can get some of these H100s.
Read the paper here ๐ Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2408.03314)
- 1 reply
posted
an
update
about 1 month ago
Post
981
๐ญ๐ฒ๐ฟ๐ผ-๐บ๐ฎ๐๐ต ๐ถ๐ป๐๐ฟ๐ผ ๐๐ผ ๐๐ ๐ต๐ถ๐๐๐ผ๐ฟ๐: ๐ณ๐ฟ๐ผ๐บ ๐๐ต๐ฒ ๐ญ๐ต๐ฑ๐ฌ๐ ๐๐ผ ๐๐ผ๐ฑ๐ฎ๐'๐ ๐๐๐ ๐ ๐
I wanted to structure my thinking about LLMs by going through their history since the 50s. This history is captivating, with the opposition between Connexionists (Rosenblatt, LeCun) and Symbolists, the first victories of "deep" neural networks, the revolution of Attention...
So I might have gone a bit too far! ๐
๐ I've made a long post summarizing the main stages of building LLMs: neural networks, optimization, backpropagation, attention layers...
โ And I've made sure to keep it 100% horrible-latex-math-free: the technical stuff is conveyed in graphs only, so it should be accessible to really anyone, even your grandfather (I'm sending it to mine right now).
Read it here in english ๐ https://aymeric-roucher.github.io/brief-history-of-ai/
Pour le post en franรงais ๐ https://aymeric-roucher.github.io/breve-histoire-de-l-ia/
I wanted to structure my thinking about LLMs by going through their history since the 50s. This history is captivating, with the opposition between Connexionists (Rosenblatt, LeCun) and Symbolists, the first victories of "deep" neural networks, the revolution of Attention...
So I might have gone a bit too far! ๐
๐ I've made a long post summarizing the main stages of building LLMs: neural networks, optimization, backpropagation, attention layers...
โ And I've made sure to keep it 100% horrible-latex-math-free: the technical stuff is conveyed in graphs only, so it should be accessible to really anyone, even your grandfather (I'm sending it to mine right now).
Read it here in english ๐ https://aymeric-roucher.github.io/brief-history-of-ai/
Pour le post en franรงais ๐ https://aymeric-roucher.github.io/breve-histoire-de-l-ia/
posted
an
update
about 2 months ago
Post
1722
๐ฆ๐๐ ๐ฎ ๐ฟ๐ฒ๐น๐ฒ๐ฎ๐๐ฒ๐ฑ: ๐ก๐ฒ๐ ๐ฆ๐ข๐ง๐ ๐ผ๐ป ๐๐ฒ๐ด๐บ๐ฒ๐ป๐๐ฎ๐๐ถ๐ผ๐ป, ๐ฏ๐ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ถ๐ป๐ด ๐๐๐ป๐๐ต๐ฒ๐๐ถ๐ฐ ๐ฑ๐ฎ๐๐ฎ ๐๐ถ๐๐ต ๐ต๐๐บ๐ฎ๐ป ๐ณ๐ฒ๐ฒ๐ฑ๐ฏ๐ฎ๐ฐ๐ธ ๐
It's a model for Object segmentation, for both image and video:
๐ input = a text prompt, or a click on a specific object
๐ output = the model draws a mask around the object. In video segmentation, the mask should follow the object's movements (it is then called a masklet)
๐ช SAM 2 is 6x faster than the previous version, it now also works on a video, and it beats SOTA by far on both image and video segmentation tasks.
How did they pull that?
The main blocker for video segmentation was that data is really hard to collect: to build your training dataset, should you manually draw masks on every frame? That would be way too costly! โก๏ธ As a result, existing video segmentation datasets have a real lack of coverage: few examples, few masklets drawn.
๐ก Key idea: researchers they decided to use a segmentation model to help them collect the dataset.
But then itโs a chicken and egg problem: you need the model to create the dataset and the opposite as well? ๐ค
โ To solve this, they build a data generation system that they scale up progressively in 3 successive manual annotations phases:
๐ฆ๐๐ฒ๐ฝ ๐ญ: Annotators use only SAM + manual editing tools on each frame โ Create 16k masklets across 1.4k videos
๐ฆ๐๐ฒ๐ฝ ๐ฎ: Then train a first SAM 2, add it in the loop to temporally propagate frames, and correct by re-doing a mask manually when an error has occured โ This gets a 5.1x speedup over data collection in phase 1! ๐ Collect 60k masklets
๐ฆ๐๐ฒ๐ฝ ๐ฏ: Now SAM 2 is more powerful, it has the โsingle clickโ prompting option, thus annotators can use it with simple clicks to re-annotate data.
They even add a completely automatic step to generate 350k more masklets!
And in turn, the model perf gradually increases.
I find this a great example of combining synthetic data generation with human annotation ๐
It's a model for Object segmentation, for both image and video:
๐ input = a text prompt, or a click on a specific object
๐ output = the model draws a mask around the object. In video segmentation, the mask should follow the object's movements (it is then called a masklet)
๐ช SAM 2 is 6x faster than the previous version, it now also works on a video, and it beats SOTA by far on both image and video segmentation tasks.
How did they pull that?
The main blocker for video segmentation was that data is really hard to collect: to build your training dataset, should you manually draw masks on every frame? That would be way too costly! โก๏ธ As a result, existing video segmentation datasets have a real lack of coverage: few examples, few masklets drawn.
๐ก Key idea: researchers they decided to use a segmentation model to help them collect the dataset.
But then itโs a chicken and egg problem: you need the model to create the dataset and the opposite as well? ๐ค
โ To solve this, they build a data generation system that they scale up progressively in 3 successive manual annotations phases:
๐ฆ๐๐ฒ๐ฝ ๐ญ: Annotators use only SAM + manual editing tools on each frame โ Create 16k masklets across 1.4k videos
๐ฆ๐๐ฒ๐ฝ ๐ฎ: Then train a first SAM 2, add it in the loop to temporally propagate frames, and correct by re-doing a mask manually when an error has occured โ This gets a 5.1x speedup over data collection in phase 1! ๐ Collect 60k masklets
๐ฆ๐๐ฒ๐ฝ ๐ฏ: Now SAM 2 is more powerful, it has the โsingle clickโ prompting option, thus annotators can use it with simple clicks to re-annotate data.
They even add a completely automatic step to generate 350k more masklets!
And in turn, the model perf gradually increases.
I find this a great example of combining synthetic data generation with human annotation ๐
posted
an
update
about 2 months ago
Post
1081
๐๐น๐ฎ๐บ๐ฎ-๐ฏ.๐ญ ๐บ๐ผ๐ฑ๐ฒ๐น๐ ๐ณ๐ถ๐ป๐ฎ๐น๐น๐ ๐ด๐ฒ๐ ๐๐ต๐ฒ๐ถ๐ฟ ๐๐ต๐ฎ๐๐ฏ๐ผ๐ ๐๐ฟ๐ฒ๐ป๐ฎ ๐ฟ๐ฎ๐ป๐ธ๐ถ๐ป๐ด ๐๏ธ
Given the impressive benchmarks published my Meta for their Llama-3.1 models, I was curious to see how these models would compare to top proprietary models on Chatbot Arena.
Now we've got the results! LMSys released the ELO derived from thousands of user votes for the new models, and here are the rankings:
๐ฅ 405B Model ranks 5th overall, in front of GPT-4-turbo! But behind GPT-4o, Claude-3.5 Sonnet and Gemini-advanced.
๐ 70B Model climbs up to 9th rank ! From 1206 โก๏ธ 1244.
๐ 8B Model improves from 1152 โก๏ธ 1170.
โ This confirms that Llama-3.1 is a good contender for any task: any of its 3 model size is much cheaper to run than equivalent proprietary models!
For instance, here are the inference prices for the top models;
โค GPT-4-Turbo inference price from OpenAI: $5/M input tokens, $15/M output tokens
โค Llama-3.1-405B from HF API (for testing only): 3$/M for input or output tokens (Source linked in the first comment)
โค Llama-3.1-405B from HF API (for testing only): free โจ
Get a head start on the HF API (resource by @andrewrreed ) ๐ https://huggingface.co/learn/cookbook/enterprise_hub_serverless_inference_api
Given the impressive benchmarks published my Meta for their Llama-3.1 models, I was curious to see how these models would compare to top proprietary models on Chatbot Arena.
Now we've got the results! LMSys released the ELO derived from thousands of user votes for the new models, and here are the rankings:
๐ฅ 405B Model ranks 5th overall, in front of GPT-4-turbo! But behind GPT-4o, Claude-3.5 Sonnet and Gemini-advanced.
๐ 70B Model climbs up to 9th rank ! From 1206 โก๏ธ 1244.
๐ 8B Model improves from 1152 โก๏ธ 1170.
โ This confirms that Llama-3.1 is a good contender for any task: any of its 3 model size is much cheaper to run than equivalent proprietary models!
For instance, here are the inference prices for the top models;
โค GPT-4-Turbo inference price from OpenAI: $5/M input tokens, $15/M output tokens
โค Llama-3.1-405B from HF API (for testing only): 3$/M for input or output tokens (Source linked in the first comment)
โค Llama-3.1-405B from HF API (for testing only): free โจ
Get a head start on the HF API (resource by @andrewrreed ) ๐ https://huggingface.co/learn/cookbook/enterprise_hub_serverless_inference_api
- 1 reply
posted
an
update
about 2 months ago
Post
2243
๐ง๐ต๐ฒ ๐ต๐๐ด๐ฒ ๐ฐ๐ผ๐๐ ๐ผ๐ณ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ผ๐ป ๐ณ๐ฟ๐ผ๐ป๐๐ถ๐ฒ๐ฟ ๐๐๐ ๐ ๐ธ
Google DeepMind recently released a great paper that shows optimal hyperparameters to train across different regimes: Scaling Exponents Across Parameterizations and Optimizers, with data from 10,000 training runs.
One engineer decided to quantify the price of such a large-scale experiment.
๐ฌ And the bill is hefty: ~13M USD
This exact number is to take with a grain of salt because many approximations were necessary to get the final result.
โ๏ธ But still this ballpark means that for this sole experiment, the price is way over what most startups or research labs could afford.
This means that open-sourcing research is more important than ever, to put everyone in the ecosystem on a roughly equal footing. Don't let OpenAI run first, they'll keep everything for themselves!
Read the full post that quantifies the paper's cost ๐ https://152334h.github.io/blog/scaling-exponents/
Google DeepMind recently released a great paper that shows optimal hyperparameters to train across different regimes: Scaling Exponents Across Parameterizations and Optimizers, with data from 10,000 training runs.
One engineer decided to quantify the price of such a large-scale experiment.
๐ฌ And the bill is hefty: ~13M USD
This exact number is to take with a grain of salt because many approximations were necessary to get the final result.
โ๏ธ But still this ballpark means that for this sole experiment, the price is way over what most startups or research labs could afford.
This means that open-sourcing research is more important than ever, to put everyone in the ecosystem on a roughly equal footing. Don't let OpenAI run first, they'll keep everything for themselves!
Read the full post that quantifies the paper's cost ๐ https://152334h.github.io/blog/scaling-exponents/
- 1 reply
posted
an
update
about 2 months ago
Post
2238
๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐๐ฎ๐๐ฎ ๐ฎ๐ป๐ฎ๐น๐๐๐: ๐ฑ๐ฟ๐ผ๐ฝ ๐๐ผ๐๐ฟ ๐ฑ๐ฎ๐๐ฎ ๐ณ๐ถ๐น๐ฒ, ๐น๐ฒ๐ ๐๐ต๐ฒ ๐๐๐ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฎ๐ป๐ฎ๐น๐๐๐ถ๐ ๐โ๏ธ
Need to make quick exploratory data analysis? โก๏ธ Get help from an agent.
I was impressed by Llama-3.1's capacity to derive insights from data. Given a csv file, it makes quick work of exploratory data analysis and can derive interesting insights.
On the data from the Kaggle titanic challenge, that records which passengers survived the Titanic wreckage, it was able by itself to derive interesting trends like "passengers that paid higher fares were more likely to survive" or "survival rate was much higher for women than men".
The cookbook even lets the agent built its own submission to the challenge, and it ranks under 3,000 out of 17,000 submissions: ๐ not bad at all!
Try it for yourself in this Space demo ๐ m-ric/agent-data-analyst
Need to make quick exploratory data analysis? โก๏ธ Get help from an agent.
I was impressed by Llama-3.1's capacity to derive insights from data. Given a csv file, it makes quick work of exploratory data analysis and can derive interesting insights.
On the data from the Kaggle titanic challenge, that records which passengers survived the Titanic wreckage, it was able by itself to derive interesting trends like "passengers that paid higher fares were more likely to survive" or "survival rate was much higher for women than men".
The cookbook even lets the agent built its own submission to the challenge, and it ranks under 3,000 out of 17,000 submissions: ๐ not bad at all!
Try it for yourself in this Space demo ๐ m-ric/agent-data-analyst
- 2 replies
Thanks a lot @Wauplin !
posted
an
update
2 months ago
Post
3194
๐๐๐ฐ ๐๐๐๐จ๐๐ข๐ง๐ ๐ญ๐๐๐ก๐ง๐ข๐ช๐ฎ๐ ๐ข๐ง ๐ญ๐ซ๐๐ง๐ฌ๐๐จ๐ซ๐ฆ๐๐ซ๐ฌ ๐ฌ๐ข๐ ๐ง๐ข๐๐ข๐๐๐ง๐ญ๐ฅ๐ฒ ๐ซ๐๐๐ฎ๐๐๐ฌ ๐ก๐๐ฅ๐ฅ๐ฎ๐๐ข๐ง๐๐ญ๐ข๐จ๐ง๐ฌ ๐
DoLa decoding, which made a conference paper at ICLR '24, has just been merged in Transformers by @joaogante and Yung-Sung Chuang.
This new decoding method is simple yet extremely impressive!
Reminder: Decoder LLMs (the GPT kind of LLM, the most common one) generate their outputs one token at a time: at each step, given a current text, they compute a logit for each token in their vocabulary that should represent the probability of this token coming next.
Then they either pick the highest logit token (greedy decoding) or sample one with a probability defined by the logits (sampling).
The authors of DoLa wanted to improve that simple method.
They knew this established fact that transformer LMs encode low-level info (like base syntax) in early layers and more high-level info like knowledge in the later layers.
๐ก This gave them their key idea: During decoding, rather than picking the token with the highest logit, ๐๐ต๐ ๐ป๐ผ๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ต๐ฒ ๐๐ผ๐ธ๐ฒ๐ป ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐บ๐ผ๐๐ ๐ถ๐บ๐ฝ๐ฟ๐ฒ๐๐๐ถ๐๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ป ๐น๐ผ๐ด๐ถ๐ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐น๐ฎ๐๐ฒ๐ฟ๐?
This gives impressive results:
๐ ๐ฑ% - ๐ฎ๐ฌ% ๐ฏ๐ฎ๐๐ฒ ๐ฝ๐ผ๐ถ๐ป๐๐ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐๐ต๐ฒ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐
๐ For instance on TruthfulQA / Open-ended, across all model sizes the increase in truthfulness is 14 base points, which is ๐ฎ๐ฟ๐ผ๐๐ป๐ฑ ๐ฐ๐ฌ% ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ๐บ๐ฒ๐ป๐ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐๐ฎ๐ป๐ฑ๐ฎ๐ฟ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด!
๐ค Wouldn't decoding take longer because of this added contrasting step? ๐ ๐ง๐ต๐ฒ ๐ฟ๐๐ป๐๐ถ๐บ๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ ๐ป๐ฒ๐ด๐น๐ถ๐ด๐ถ๐ฏ๐น๐ฒ, ๐ญ ๐๐ผ ๐ด% ๐ผ๐ป๐น๐.
Paper added to my collection ๐ m-ric/optimization-mechanics-661d543a5fc6ca1dc84284a0
DoLa decoding, which made a conference paper at ICLR '24, has just been merged in Transformers by @joaogante and Yung-Sung Chuang.
This new decoding method is simple yet extremely impressive!
Reminder: Decoder LLMs (the GPT kind of LLM, the most common one) generate their outputs one token at a time: at each step, given a current text, they compute a logit for each token in their vocabulary that should represent the probability of this token coming next.
Then they either pick the highest logit token (greedy decoding) or sample one with a probability defined by the logits (sampling).
The authors of DoLa wanted to improve that simple method.
They knew this established fact that transformer LMs encode low-level info (like base syntax) in early layers and more high-level info like knowledge in the later layers.
๐ก This gave them their key idea: During decoding, rather than picking the token with the highest logit, ๐๐ต๐ ๐ป๐ผ๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ต๐ฒ ๐๐ผ๐ธ๐ฒ๐ป ๐๐ถ๐๐ต ๐๐ต๐ฒ ๐บ๐ผ๐๐ ๐ถ๐บ๐ฝ๐ฟ๐ฒ๐๐๐ถ๐๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ป ๐น๐ผ๐ด๐ถ๐ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐น๐ฎ๐๐ฒ๐ฟ๐?
This gives impressive results:
๐ ๐ฑ% - ๐ฎ๐ฌ% ๐ฏ๐ฎ๐๐ฒ ๐ฝ๐ผ๐ถ๐ป๐๐ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ฎ๐ฐ๐ฟ๐ผ๐๐ ๐๐ต๐ฒ ๐ฏ๐ฒ๐ป๐ฐ๐ต๐บ๐ฎ๐ฟ๐ธ๐
๐ For instance on TruthfulQA / Open-ended, across all model sizes the increase in truthfulness is 14 base points, which is ๐ฎ๐ฟ๐ผ๐๐ป๐ฑ ๐ฐ๐ฌ% ๐ถ๐บ๐ฝ๐ฟ๐ผ๐๐ฒ๐บ๐ฒ๐ป๐ ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐๐ฎ๐ป๐ฑ๐ฎ๐ฟ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด!
๐ค Wouldn't decoding take longer because of this added contrasting step? ๐ ๐ง๐ต๐ฒ ๐ฟ๐๐ป๐๐ถ๐บ๐ฒ ๐ถ๐ป๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ ๐ถ๐ ๐ป๐ฒ๐ด๐น๐ถ๐ด๐ถ๐ฏ๐น๐ฒ, ๐ญ ๐๐ผ ๐ด% ๐ผ๐ป๐น๐.
Paper added to my collection ๐ m-ric/optimization-mechanics-661d543a5fc6ca1dc84284a0
- 2 replies
replied to
their
post
2 months ago
You can use this newer repo in Github @sidbin : https://github.com/aymeric-roucher/GAIA, it has a requirements.txt!
posted
an
update
2 months ago
Post
849
One more cookbook:
Agent for self-correcting Text-to-SQL ๐งโ๐ป
What if the query generated by your Text-to-SQL pipeline is correct SQL but returns wrong results?
๐ We need to add a critique step
โ That's very simple with an agent!
Check out the notebook! ๐
https://huggingface.co/learn/cookbook/agent_text_to_sql
Agent for self-correcting Text-to-SQL ๐งโ๐ป
What if the query generated by your Text-to-SQL pipeline is correct SQL but returns wrong results?
๐ We need to add a critique step
โ That's very simple with an agent!
Check out the notebook! ๐
https://huggingface.co/learn/cookbook/agent_text_to_sql
posted
an
update
2 months ago
Post
1847
New cookbook!
I show to to make agentic RAG using Transformers Agents.
Compared to vanilla RAG, agentic RAG can:
โ Reformulate the query
โ Critique the retrived content to re-retrieve if needed
โก๏ธ Score increase of 8.5%! ๐ช (Llama-3-70B-judge)
Read it here ๐ https://huggingface.co/learn/cookbook/agent_rag
I show to to make agentic RAG using Transformers Agents.
Compared to vanilla RAG, agentic RAG can:
โ Reformulate the query
โ Critique the retrived content to re-retrieve if needed
โก๏ธ Score increase of 8.5%! ๐ช (Llama-3-70B-judge)
Read it here ๐ https://huggingface.co/learn/cookbook/agent_rag
replied to
their
post
3 months ago
It's not using GPT-4o for evaluation, evaluation is done with exact string match!
posted
an
update
3 months ago
Post
2647
๐ฌ๐ผ๐ ๐ฑ๐ผ๐ป'๐ ๐ป๐ฒ๐ฒ๐ฑ "๐ณ๐๐ป๐ฐ๐๐ถ๐ผ๐ป ๐ฐ๐ฎ๐น๐น๐ถ๐ป๐ด ๐ณ๐ถ๐ป๐ฒ-๐๐๐ป๐ถ๐ป๐ด" ๐๐ผ ๐ฏ๐๐ถ๐น๐ฑ ๐ด๐ผ๐ผ๐ฑ ๐ฎ๐ด๐ฒ๐ป๐๐ โ
It's trendy to share models "fine-tuned for function calling"; but from my observations, this fine-tuning is not necessary or sufficient to build good agent systems.
To name only a few:
๐ฆโโฌ Nexusflow/๐ก๐ฒ๐ ๐๐๐ฅ๐ฎ๐๐ฒ๐ป-๐ฉ๐ฎ-๐ญ๐ฏ๐
โ CohereForAI/๐ฐ๐ฐ๐ฎ๐ถ-๐ฐ๐ผ๐บ๐บ๐ฎ๐ป๐ฑ-๐ฟ-๐ฝ๐น๐๐
โต๏ธ mistralai/๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น-๐ด๐ ๐ฎ๐ฎ๐-๐๐ป๐๐๐ฟ๐๐ฐ๐-๐๐ฌ.๐ญ
"Fine-tuned for function-calling" generally means "fine-tuned to generate function calls in correct JSON for extremely simple tasks". In other terms, it means "improve the formatting of the tool calls".
Yet I discovered two things while improving Transformers Agents:
๐ง Even when used as JSON agents, these fine-tuned models don't perform very well
๐ ๐๐ค๐ค๐ ๐๐๐จ๐ ๐ข๐ค๐๐๐ก๐จ ๐ฅ๐๐ง๐๐ค๐ง๐ข ๐๐๐ฉ๐ฉ๐๐ง ๐ฌ๐๐ฉ๐๐ค๐ช๐ฉ ๐๐ฃ๐ฎ ๐๐๐ฃ๐-๐ฉ๐ช๐ฃ๐๐ฃ๐, ๐๐ช๐จ๐ฉ ๐ฅ๐ก๐๐๐ฃ ๐ฅ๐ง๐ค๐ข๐ฅ๐ฉ๐๐ฃ๐. (Llama-3-70B-Instruct, GPT-4o, Claude-3.5-Sonnet)
๐ The graph below shows the count of errors for my GPT-4o validation run on the GAIA benchmark: ๐ฐ๐๐๐๐๐ฟ๐๐๐๐๐๐๐ด๐๐๐๐ and ๐ฐ๐๐๐๐๐ด๐ก๐๐๐๐๐๐๐๐ด๐๐๐๐ are the ones caused by incorrect formatting.
โค As you can see, their count is already close to 0!
And given that GPT-4o is certainly not fine-tuned for our Code tool calling format, this shows that "function calling fine-tuning" is not necessary!
The hardest thing to get right in an agent is still to ๐ฅ๐ก๐๐ฃ ๐๐ค๐ค๐ ๐ฉ๐๐จ๐ -๐จ๐ค๐ก๐ซ๐๐ฃ๐ ๐ฉ๐ง๐๐๐๐๐ฉ๐ค๐ง๐๐๐จ ๐ค๐ซ๐๐ง ๐จ๐๐ซ๐๐ง๐๐ก ๐จ๐ฉ๐๐ฅ๐จ.
To improve this, we could:
- Use more powerful base models
- Make tool calling datasets with complex solving trajectories
- Use RL! cc @lvwerra
It's trendy to share models "fine-tuned for function calling"; but from my observations, this fine-tuning is not necessary or sufficient to build good agent systems.
To name only a few:
๐ฆโโฌ Nexusflow/๐ก๐ฒ๐ ๐๐๐ฅ๐ฎ๐๐ฒ๐ป-๐ฉ๐ฎ-๐ญ๐ฏ๐
โ CohereForAI/๐ฐ๐ฐ๐ฎ๐ถ-๐ฐ๐ผ๐บ๐บ๐ฎ๐ป๐ฑ-๐ฟ-๐ฝ๐น๐๐
โต๏ธ mistralai/๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น-๐ด๐ ๐ฎ๐ฎ๐-๐๐ป๐๐๐ฟ๐๐ฐ๐-๐๐ฌ.๐ญ
"Fine-tuned for function-calling" generally means "fine-tuned to generate function calls in correct JSON for extremely simple tasks". In other terms, it means "improve the formatting of the tool calls".
Yet I discovered two things while improving Transformers Agents:
๐ง Even when used as JSON agents, these fine-tuned models don't perform very well
๐ ๐๐ค๐ค๐ ๐๐๐จ๐ ๐ข๐ค๐๐๐ก๐จ ๐ฅ๐๐ง๐๐ค๐ง๐ข ๐๐๐ฉ๐ฉ๐๐ง ๐ฌ๐๐ฉ๐๐ค๐ช๐ฉ ๐๐ฃ๐ฎ ๐๐๐ฃ๐-๐ฉ๐ช๐ฃ๐๐ฃ๐, ๐๐ช๐จ๐ฉ ๐ฅ๐ก๐๐๐ฃ ๐ฅ๐ง๐ค๐ข๐ฅ๐ฉ๐๐ฃ๐. (Llama-3-70B-Instruct, GPT-4o, Claude-3.5-Sonnet)
๐ The graph below shows the count of errors for my GPT-4o validation run on the GAIA benchmark: ๐ฐ๐๐๐๐๐ฟ๐๐๐๐๐๐๐ด๐๐๐๐ and ๐ฐ๐๐๐๐๐ด๐ก๐๐๐๐๐๐๐๐ด๐๐๐๐ are the ones caused by incorrect formatting.
โค As you can see, their count is already close to 0!
And given that GPT-4o is certainly not fine-tuned for our Code tool calling format, this shows that "function calling fine-tuning" is not necessary!
The hardest thing to get right in an agent is still to ๐ฅ๐ก๐๐ฃ ๐๐ค๐ค๐ ๐ฉ๐๐จ๐ -๐จ๐ค๐ก๐ซ๐๐ฃ๐ ๐ฉ๐ง๐๐๐๐๐ฉ๐ค๐ง๐๐๐จ ๐ค๐ซ๐๐ง ๐จ๐๐ซ๐๐ง๐๐ก ๐จ๐ฉ๐๐ฅ๐จ.
To improve this, we could:
- Use more powerful base models
- Make tool calling datasets with complex solving trajectories
- Use RL! cc @lvwerra
-
- 3 replies
posted
an
update
3 months ago
Post
767
๐๐ซ๐๐ง๐ฌ๐๐จ๐ซ๐ฆ๐๐ซ๐ฌ ๐๐ ๐๐ง๐ญ๐ฌ ๐ซ๐๐๐๐ก๐๐ฌ ๐ญ๐ก๐ ๐ญ๐จ๐ฉ ๐จ๐ ๐๐๐๐ ๐ฅ๐๐๐๐๐ซ๐๐จ๐๐ซ๐! ๐ฅณ
We've been improving Transformers Agents a lot lately.
So with @sergeipetrov we set out to prove that it's the best agent framework out there.
To prove this, we went to beat the ๐๐๐๐ ๐น๐ฒ๐ฎ๐ฑ๐ฒ๐ฟ๐ฏ๐ผ๐ฎ๐ฟ๐ฑ, the most comprehensive benchmark out there for evaluating LLM agents.
Its questions make you explore different flavours of pain:
๐ ๏ธ ๐ฅ๐ฒ๐พ๐๐ถ๐ฟ๐ฒ ๐๐๐ถ๐ป๐ด ๐๐ผ๐ผ๐น๐, at least a web browser
๐ข ๐ฅ๐ถ๐ด๐ผ๐ฟ๐ผ๐๐ ๐น๐ผ๐ด๐ถ๐ฐ, many questions having strong math aspects
๐ผ๏ธ ๐ ๐๐น๐๐ถ๐บ๐ผ๐ฑ๐ฎ๐น, the agent had to handle all file types: ๐, ๐ผ๏ธ, ๐ฌ...
๐ฃ ๐ ๐๐น๐๐ถ-๐๐๐ฒ๐ฝ, with many questions requiring over 10 steps to be solved.
Some Level 3 questions are crazy hard ๐ณ
> "In NASAโs Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute?"
(๐ฏ๐ฐ ๐ง๐ช๐ญ๐ฆ ๐ข๐ต๐ต๐ข๐ค๐ฉ๐ฆ๐ฅ ๐ฐ๐ง ๐ค๐ฐ๐ถ๐ณ๐ด๐ฆ, ๐ต๐ฉ๐ฆ ๐ข๐จ๐ฆ๐ฏ๐ต ๐ฉ๐ข๐ด ๐ต๐ฐ ๐ง๐ช๐ฏ๐ฅ ๐ข๐ญ๐ญ ๐ต๐ฉ๐ฆ ๐ช๐ฏ๐ง๐ฐ)
โก๏ธ We used Transformers Agents' React Code Agent, that writes its actions in code. We created a new planning component that we'll incorporate in the framework. More info soon in a blog post!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Our submission scores #2 overall on the test set and #1 on the validation set. On both sets we're the leading submission based on a public framework, beating Microsoft's Autogen.
๐ฅ On both sets we are #1 on the hardest Level 3 questions, reaching nearly 20%.
๐๐ค ๐๐๐๐๐ ๐ค๐ช๐ฉ ๐ฉ๐๐ ๐ก๐๐๐๐๐ง๐๐ค๐๐ง๐ ๐ gaia-benchmark/leaderboard
We've been improving Transformers Agents a lot lately.
So with @sergeipetrov we set out to prove that it's the best agent framework out there.
To prove this, we went to beat the ๐๐๐๐ ๐น๐ฒ๐ฎ๐ฑ๐ฒ๐ฟ๐ฏ๐ผ๐ฎ๐ฟ๐ฑ, the most comprehensive benchmark out there for evaluating LLM agents.
Its questions make you explore different flavours of pain:
๐ ๏ธ ๐ฅ๐ฒ๐พ๐๐ถ๐ฟ๐ฒ ๐๐๐ถ๐ป๐ด ๐๐ผ๐ผ๐น๐, at least a web browser
๐ข ๐ฅ๐ถ๐ด๐ผ๐ฟ๐ผ๐๐ ๐น๐ผ๐ด๐ถ๐ฐ, many questions having strong math aspects
๐ผ๏ธ ๐ ๐๐น๐๐ถ๐บ๐ผ๐ฑ๐ฎ๐น, the agent had to handle all file types: ๐, ๐ผ๏ธ, ๐ฌ...
๐ฃ ๐ ๐๐น๐๐ถ-๐๐๐ฒ๐ฝ, with many questions requiring over 10 steps to be solved.
Some Level 3 questions are crazy hard ๐ณ
> "In NASAโs Astronomy Picture of the Day on 2006 January 21, two astronauts are visible, with one appearing much smaller than the other. As of August 2023, out of the astronauts in the NASA Astronaut Group that the smaller astronaut was a member of, which one spent the least time in space, and how many minutes did he spend in space, rounded to the nearest minute?"
(๐ฏ๐ฐ ๐ง๐ช๐ญ๐ฆ ๐ข๐ต๐ต๐ข๐ค๐ฉ๐ฆ๐ฅ ๐ฐ๐ง ๐ค๐ฐ๐ถ๐ณ๐ด๐ฆ, ๐ต๐ฉ๐ฆ ๐ข๐จ๐ฆ๐ฏ๐ต ๐ฉ๐ข๐ด ๐ต๐ฐ ๐ง๐ช๐ฏ๐ฅ ๐ข๐ญ๐ญ ๐ต๐ฉ๐ฆ ๐ช๐ฏ๐ง๐ฐ)
โก๏ธ We used Transformers Agents' React Code Agent, that writes its actions in code. We created a new planning component that we'll incorporate in the framework. More info soon in a blog post!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Our submission scores #2 overall on the test set and #1 on the validation set. On both sets we're the leading submission based on a public framework, beating Microsoft's Autogen.
๐ฅ On both sets we are #1 on the hardest Level 3 questions, reaching nearly 20%.
๐๐ค ๐๐๐๐๐ ๐ค๐ช๐ฉ ๐ฉ๐๐ ๐ก๐๐๐๐๐ง๐๐ค๐๐ง๐ ๐ gaia-benchmark/leaderboard
-
- 2 replies
replied to
their
post
3 months ago
Great idea! Can I build it @victor or you'd like to make it yourself?
posted
an
update
3 months ago
Post
3115
๐ฐ ๐๐ฒ๐ ๐๐ต๐ฒ ๐ฝ๐ฟ๐ถ๐ฐ๐ฒ ๐ผ๐ณ ๐ฎ๐ป๐ ๐๐๐ ๐๐ฃ๐ ๐ฟ๐ฒ๐พ๐๐ฒ๐๐ โ ๐๐ผ๐ธ๐ฒ๐ป๐ฐ๐ผ๐๐
I've just found out about ๐ฐ๐๐๐๐๐พ๐๐-๐ฐ๐ธ/๐๐๐๐๐๐๐๐๐ (https://github.com/AgentOps-AI/tokencost).
๐ง๐ต๐ถ๐ ๐น๐ถ๐ฏ๐ฟ๐ฎ๐ฟ๐ ๐ด๐ถ๐๐ฒ๐ ๐๐ผ๐ ๐๐ต๐ฒ ๐ฝ๐ฟ๐ถ๐ฐ๐ฒ ๐ผ๐ณ ๐๐ผ๐๐ฟ ๐ฐ๐ฎ๐น๐น๐ ๐๐ผ ๐ฎ๐ป๐ ๐๐๐ ๐๐ฃ๐: OpenAI, Anthropic, Mistral, AWS or Databricks...
For any model, you can use as input either string prompts or messages, and get as outputs either the price or token count.
Congrats to the AgentOps-AI team: this will be very useful when trying to get a ballpark estimate of a project's price, to compare APIs, or for precise monitoring of usage!
โจ Daily reminder: ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ฎ๐ป ๐๐ญ๐ฌ๐ฌ ๐ฐ๐ผ๐๐๐ ๐๐ผ๐ ๐ฒ๐ ๐ฎ๐ฐ๐๐น๐ $๐ฌ.๐ฌ๐ฌ/๐ต๐ผ๐๐ฟ (or 0.00โฌ in current exchange rates) on a HF space with ZeroGPU!
Learn more on ZeroGPU ๐ https://www.datacenterdynamics.com/en/news/hugging-face-launches-zerogpu-project-to-democratize-ai-gives-away-10-million-worth-of-compute/
I've just found out about ๐ฐ๐๐๐๐๐พ๐๐-๐ฐ๐ธ/๐๐๐๐๐๐๐๐๐ (https://github.com/AgentOps-AI/tokencost).
๐ง๐ต๐ถ๐ ๐น๐ถ๐ฏ๐ฟ๐ฎ๐ฟ๐ ๐ด๐ถ๐๐ฒ๐ ๐๐ผ๐ ๐๐ต๐ฒ ๐ฝ๐ฟ๐ถ๐ฐ๐ฒ ๐ผ๐ณ ๐๐ผ๐๐ฟ ๐ฐ๐ฎ๐น๐น๐ ๐๐ผ ๐ฎ๐ป๐ ๐๐๐ ๐๐ฃ๐: OpenAI, Anthropic, Mistral, AWS or Databricks...
For any model, you can use as input either string prompts or messages, and get as outputs either the price or token count.
Congrats to the AgentOps-AI team: this will be very useful when trying to get a ballpark estimate of a project's price, to compare APIs, or for precise monitoring of usage!
โจ Daily reminder: ๐ฟ๐๐ป๐ป๐ถ๐ป๐ด ๐ฎ๐ป ๐๐ญ๐ฌ๐ฌ ๐ฐ๐ผ๐๐๐ ๐๐ผ๐ ๐ฒ๐ ๐ฎ๐ฐ๐๐น๐ $๐ฌ.๐ฌ๐ฌ/๐ต๐ผ๐๐ฟ (or 0.00โฌ in current exchange rates) on a HF space with ZeroGPU!
Learn more on ZeroGPU ๐ https://www.datacenterdynamics.com/en/news/hugging-face-launches-zerogpu-project-to-democratize-ai-gives-away-10-million-worth-of-compute/
-
- 5 replies
posted
an
update
4 months ago
Post
1840
๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐ฎ๐ป ๐ฎ๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐๐ผ๐ฟ๐ธ๐ณ๐น๐ผ๐ ๐๐๐ฒ ๐ถ๐๐ ๐๐๐ ๐ฒ๐ป๐ด๐ถ๐ป๐ฒ ๐๐ผ ๐๐ผ๐น๐๐ฒ ๐๐ฎ๐๐ธ๐?
โก๏ธ I made my first ever ๐ฎ๐ข๐ฏ๐ช๐ฎ video to show just that:
๐ช๐ฎ๐๐ฐ๐ต ๐ฏ๐ฒ๐น๐ผ๐ ๐ต๐ผ๐ ๐ฎ ๐ฅ๐ฒ๐ฎ๐ฐ๐ ๐๐ด๐ฒ๐ป๐ ๐๐ผ๐น๐๐ฒ๐ ๐ฎ ๐๐ถ๐บ๐ฝ๐น๐ฒ ๐๐ฎ๐๐ธ, by leveraging its memory to iterate on previous actions! ๐ฌ๐
Read our blog post on Agents: https://huggingface.co/blog/agents
โก๏ธ I made my first ever ๐ฎ๐ข๐ฏ๐ช๐ฎ video to show just that:
๐ช๐ฎ๐๐ฐ๐ต ๐ฏ๐ฒ๐น๐ผ๐ ๐ต๐ผ๐ ๐ฎ ๐ฅ๐ฒ๐ฎ๐ฐ๐ ๐๐ด๐ฒ๐ป๐ ๐๐ผ๐น๐๐ฒ๐ ๐ฎ ๐๐ถ๐บ๐ฝ๐น๐ฒ ๐๐ฎ๐๐ธ, by leveraging its memory to iterate on previous actions! ๐ฌ๐
Read our blog post on Agents: https://huggingface.co/blog/agents
- 1 reply
posted
an
update
4 months ago
Post
831
๐๐ง๐๐ฉ๐๐ฃ๐ ๐ฉ๐ค๐ค๐ก ๐๐๐ก๐ก๐จ ๐๐ฃ ๐๐ค๐๐ ๐๐ช๐จ๐ฉ ๐ฌ๐ค๐ง๐ ๐จ ๐๐๐ฉ๐ฉ๐๐ง ๐ฉ๐๐๐ฃ ๐
๐๐๐ ๐ช
I was really happy to learn today by @sergeipetrov that paper ๐๐น๐ฆ๐ค๐ถ๐ต๐ข๐ฃ๐ญ๐ฆ ๐๐ฐ๐ฅ๐ฆ ๐๐ค๐ต๐ช๐ฐ๐ฏ๐ด ๐๐ญ๐ช๐ค๐ช๐ต ๐๐ฆ๐ต๐ต๐ฆ๐ณ ๐๐๐ ๐๐จ๐ฆ๐ฏ๐ต๐ด was accepted at ICLR 2024!
As a reminder, an agent is a system in which you embed a LLM engine, to let it call tools.
These tools are meant like an IronMan suit, to supplement the LLM in areas that it isn't good at.
๐งโ๐ป For instance your friendly LLM may be terrible at calculating powers of floating numbers ("What is X ^0.2947 ?"), so it should use a calculator.
๐It may be terrible at knowing precise facts ("What was the date of the Golden Bull?") so it should use a web browser.
So the agent system will prompt an agent with "Now you can use these tools: calculator, search,..."
But ๐๐ค๐ฌ ๐จ๐๐ค๐ช๐ก๐ ๐ฉ๐๐ ๐๐๐๐ฃ๐ฉ ๐๐ญ๐ฅ๐ง๐๐จ๐จ ๐๐ฉ๐จ ๐๐๐ฉ๐๐ค๐ฃ๐จ?
All well known frameworks let agents write their actions as JSON strings.
We ๐ฝ๐ฟ๐ฒ๐ณ๐ฒ๐ฟ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐ด๐ผ ๐๐ถ๐๐ต ๐ณ๐ผ๐ฟ๐บ๐๐น๐ฎ๐๐ถ๐ป๐ด ๐ฎ๐ฐ๐๐ถ๐ผ๐ป๐ ๐ถ๐ป ๐๐ผ๐ฑ๐ฒ, ๐๐ต๐ถ๐ฐ๐ต ๐ถ๐ ๐บ๐๐ฐ๐ต ๐บ๐ผ๐ฟ๐ฒ ๐๐ฒ๐ฟ๐๐ฎ๐๐ถ๐น๐ฒ ๐ฎ๐ป๐ฑ ๐ฐ๐ผ๐ป๐ฐ๐ถ๐๐ฒ, ๐ฎ๐ป๐ฑ ๐ฎ๐น๐น๐ผ๐๐ ๐๐ผ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ฎ๐ฐ๐๐ถ๐ผ๐ป๐ ๐๐ฒ๐ฎ๐บ๐น๐ฒ๐๐๐น๐: see the picture attached for an example where Code formulation really shines.
And the paper confirms our choice: researchers show that ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐ฆ๐ข๐ก ๐ผ๐ฟ ๐ฝ๐น๐ฎ๐ถ๐ป ๐๐ฒ๐ ๐, ๐๐ผ๐ฑ๐ฒ ๐ถ๐ ๐ฏ๐ฒ๐๐๐ฒ๐ฟ ๐ฏ๐ผ๐๐ต ๐ถ๐ป ๐ฐ๐ผ๐ป๐ฐ๐ถ๐๐ฒ๐ป๐ฒ๐๐ ๐ฎ๐ป๐ฑ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ:
โค Up to 30% fewer steps for the same actions (much more concise)
โค Up to 20% higher performance on benchmarks
And we find additional benefits, for instance a natural handling of variables.
Read the paper here ๐ Executable Code Actions Elicit Better LLM Agents (2402.01030)
Get your ReactCodeAgent running with our Agents framework! ๐ https://huggingface.co/learn/cookbook/agents
I was really happy to learn today by @sergeipetrov that paper ๐๐น๐ฆ๐ค๐ถ๐ต๐ข๐ฃ๐ญ๐ฆ ๐๐ฐ๐ฅ๐ฆ ๐๐ค๐ต๐ช๐ฐ๐ฏ๐ด ๐๐ญ๐ช๐ค๐ช๐ต ๐๐ฆ๐ต๐ต๐ฆ๐ณ ๐๐๐ ๐๐จ๐ฆ๐ฏ๐ต๐ด was accepted at ICLR 2024!
As a reminder, an agent is a system in which you embed a LLM engine, to let it call tools.
These tools are meant like an IronMan suit, to supplement the LLM in areas that it isn't good at.
๐งโ๐ป For instance your friendly LLM may be terrible at calculating powers of floating numbers ("What is X ^0.2947 ?"), so it should use a calculator.
๐It may be terrible at knowing precise facts ("What was the date of the Golden Bull?") so it should use a web browser.
So the agent system will prompt an agent with "Now you can use these tools: calculator, search,..."
But ๐๐ค๐ฌ ๐จ๐๐ค๐ช๐ก๐ ๐ฉ๐๐ ๐๐๐๐ฃ๐ฉ ๐๐ญ๐ฅ๐ง๐๐จ๐จ ๐๐ฉ๐จ ๐๐๐ฉ๐๐ค๐ฃ๐จ?
All well known frameworks let agents write their actions as JSON strings.
We ๐ฝ๐ฟ๐ฒ๐ณ๐ฒ๐ฟ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐ด๐ผ ๐๐ถ๐๐ต ๐ณ๐ผ๐ฟ๐บ๐๐น๐ฎ๐๐ถ๐ป๐ด ๐ฎ๐ฐ๐๐ถ๐ผ๐ป๐ ๐ถ๐ป ๐๐ผ๐ฑ๐ฒ, ๐๐ต๐ถ๐ฐ๐ต ๐ถ๐ ๐บ๐๐ฐ๐ต ๐บ๐ผ๐ฟ๐ฒ ๐๐ฒ๐ฟ๐๐ฎ๐๐ถ๐น๐ฒ ๐ฎ๐ป๐ฑ ๐ฐ๐ผ๐ป๐ฐ๐ถ๐๐ฒ, ๐ฎ๐ป๐ฑ ๐ฎ๐น๐น๐ผ๐๐ ๐๐ผ ๐ฐ๐ต๐ฎ๐ถ๐ป ๐ฎ๐ฐ๐๐ถ๐ผ๐ป๐ ๐๐ฒ๐ฎ๐บ๐น๐ฒ๐๐๐น๐: see the picture attached for an example where Code formulation really shines.
And the paper confirms our choice: researchers show that ๐ฐ๐ผ๐บ๐ฝ๐ฎ๐ฟ๐ฒ๐ฑ ๐๐ผ ๐๐ฆ๐ข๐ก ๐ผ๐ฟ ๐ฝ๐น๐ฎ๐ถ๐ป ๐๐ฒ๐ ๐, ๐๐ผ๐ฑ๐ฒ ๐ถ๐ ๐ฏ๐ฒ๐๐๐ฒ๐ฟ ๐ฏ๐ผ๐๐ต ๐ถ๐ป ๐ฐ๐ผ๐ป๐ฐ๐ถ๐๐ฒ๐ป๐ฒ๐๐ ๐ฎ๐ป๐ฑ ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ:
โค Up to 30% fewer steps for the same actions (much more concise)
โค Up to 20% higher performance on benchmarks
And we find additional benefits, for instance a natural handling of variables.
Read the paper here ๐ Executable Code Actions Elicit Better LLM Agents (2402.01030)
Get your ReactCodeAgent running with our Agents framework! ๐ https://huggingface.co/learn/cookbook/agents
posted
an
update
4 months ago
Post
992
๐๐๐ฐ ๐ ๐ฎ๐ข๐๐ ๐ข๐ง ๐จ๐ฎ๐ซ ๐๐ฉ๐๐ง-๐๐จ๐ฎ๐ซ๐๐ ๐๐ ๐๐จ๐จ๐ค๐๐จ๐จ๐ค: ๐๐ฉ๐ง๐ช๐๐ฉ๐ช๐ง๐๐ ๐๐๐ฃ๐๐ง๐๐ฉ๐๐ค๐ฃ! โจ
Many use LLM use cases involve generating outputs with a specific structure.
โก๏ธ For instance when using an LLM as a judge to evaluate another model's outputs, you need it to give you not only a score, but also the rationale for this score, and maybe a confidence level.
So you do not need only "score: 1", but more a dictionary like:
๐ค How to force your LLM to generate such a structured output?
๐๏ธ ๐๐ผ๐ป๐๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด is a great technique to generate structured output: you can specify a grammar (=set of rules) that the output should follow, and ๐ฐ๐ผ๐ป๐๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐๐ต๐ฒ๐ป ๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐ ๐๐ต๐ฒ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ๐ฟ ๐๐ผ ๐ผ๐ป๐น๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ผ๐ธ๐ฒ๐ป๐ ๐๐ต๐ฎ๐ ๐ฟ๐ฒ๐๐ฝ๐ฒ๐ฐ๐ ๐๐ผ๐๐ฟ ๐ด๐ฟ๐ฎ๐บ๐บ๐ฎ๐ฟ.
I've created a guide to show you how to use it, both via our Inference API and locally using ๐ฐ๐ถ๐ต๐ญ๐ช๐ฏ๐ฆ๐ด!
๐ Read it here: https://huggingface.co/learn/cookbook/structured_generation
Thank you @stevhliu for your great help in improving it!
Many use LLM use cases involve generating outputs with a specific structure.
โก๏ธ For instance when using an LLM as a judge to evaluate another model's outputs, you need it to give you not only a score, but also the rationale for this score, and maybe a confidence level.
So you do not need only "score: 1", but more a dictionary like:
{
"rationale": "The answer does not match the true answer at all."
"score": 1,
"confidence_level": 0.85
}๐ค How to force your LLM to generate such a structured output?
๐๏ธ ๐๐ผ๐ป๐๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด is a great technique to generate structured output: you can specify a grammar (=set of rules) that the output should follow, and ๐ฐ๐ผ๐ป๐๐๐ฟ๐ฎ๐ถ๐ป๐ฒ๐ฑ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐๐ต๐ฒ๐ป ๐ณ๐ผ๐ฟ๐ฐ๐ฒ๐ ๐๐ต๐ฒ ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ฒ๐ฟ ๐๐ผ ๐ผ๐ป๐น๐ ๐ฝ๐ถ๐ฐ๐ธ ๐๐ผ๐ธ๐ฒ๐ป๐ ๐๐ต๐ฎ๐ ๐ฟ๐ฒ๐๐ฝ๐ฒ๐ฐ๐ ๐๐ผ๐๐ฟ ๐ด๐ฟ๐ฎ๐บ๐บ๐ฎ๐ฟ.
I've created a guide to show you how to use it, both via our Inference API and locally using ๐ฐ๐ถ๐ต๐ญ๐ช๐ฏ๐ฆ๐ด!
๐ Read it here: https://huggingface.co/learn/cookbook/structured_generation
Thank you @stevhliu for your great help in improving it!
posted
an
update
5 months ago
Post
2759
๐ฐโ ๐๐๐ฌ๐๐๐ซ๐๐ก ๐๐จ๐ซ ๐ญ๐ก๐ ๐ฏ๐๐ซ๐ฒ ๐๐๐ ๐๐จ๐จ๐ซ - ๐๐๐๐ฅ๐ข๐ง๐ ๐ฅ๐๐ฐ๐ฌ ๐ซ๐๐ฉ๐ฅ๐ข๐๐๐ญ๐ข๐จ๐ง
๐ Good news: ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ ๐ฐ๐๐๐๐ถ๐ป๐ด-๐ฒ๐ฑ๐ด๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ถ๐๐ต ๐ฎ ๐ฐ๐ฎ๐น๐ฐ๐๐น๐ฎ๐๐ผ๐ฟ ๐ฎ๐ป๐ฑ ๐ ๐ถ๐ฐ๐ฟ๐ผ๐๐ผ๐ณ๐ ๐ฃ๐ฎ๐ถ๐ป๐ ๐ฎ๐ฌ๐ฌ๐ฒ!
The Chinchilla experiments (by Google DeepMind) ran hundreds of pre-trainings with models >1B parameters (I do not want to imagine how much that cost) to ๐ณ๐ถ๐ป๐ฑ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐บ๐ฎ๐น ๐ฟ๐ฎ๐๐ถ๐ผ ๐ผ๐ณ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ฒ ๐๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐๐ผ๐ธ๐ฒ๐ป๐. Why is this question so important?
Well, you only ever have access to a fixed compute, counted in FLOPs (floating point operations). So if your model is bigger, you will have less compute to train on many tokens, and if you want to train on more tokens, your model will be smaller. When model trainings cost million, you absolutely need to get this right.
The new paper "Chinchilla Scaling: A replication attempt" by Epoch AI sets on on the ambitious goal of reproducing this.
But since the authors do not have infinite money, they decided to directly run their computations from DeepMind's own experiments! They took the figure from the last experiment (cf slide below), measured point positions, picked color codes, and ended up reconstructing the underlying data.
๐ฅ They then just fit the scaling laws proposed by the Chinchilla Authors, but arrived at wildly different results! They find that as a rough rule of thumb, you should use 20 training tokens for each parameter in your model, instead of the 70 obtained in the original paper. They also point out inconsistencies in the paper, and unrealistically narrow confidence intervals.
โก๏ธ This only contradicts the results from the last (out of 3) experiments in the Chinchilla paper. And the model trained at the end of the Chinchilla paper still seems properly scaled.
โ But it does show that a tiny bit more theoretical work can go a long way, especially given the huge financial costs that such an error can have!
๐ Good news: ๐๐ผ๐ ๐ฐ๐ฎ๐ป ๐ฑ๐ผ ๐ฐ๐๐๐๐ถ๐ป๐ด-๐ฒ๐ฑ๐ด๐ฒ ๐ฟ๐ฒ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐๐ถ๐๐ต ๐ฎ ๐ฐ๐ฎ๐น๐ฐ๐๐น๐ฎ๐๐ผ๐ฟ ๐ฎ๐ป๐ฑ ๐ ๐ถ๐ฐ๐ฟ๐ผ๐๐ผ๐ณ๐ ๐ฃ๐ฎ๐ถ๐ป๐ ๐ฎ๐ฌ๐ฌ๐ฒ!
The Chinchilla experiments (by Google DeepMind) ran hundreds of pre-trainings with models >1B parameters (I do not want to imagine how much that cost) to ๐ณ๐ถ๐ป๐ฑ ๐๐ต๐ฒ ๐ผ๐ฝ๐๐ถ๐บ๐ฎ๐น ๐ฟ๐ฎ๐๐ถ๐ผ ๐ผ๐ณ ๐บ๐ผ๐ฑ๐ฒ๐น ๐๐ถ๐๐ฒ ๐๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐๐ผ๐ธ๐ฒ๐ป๐. Why is this question so important?
Well, you only ever have access to a fixed compute, counted in FLOPs (floating point operations). So if your model is bigger, you will have less compute to train on many tokens, and if you want to train on more tokens, your model will be smaller. When model trainings cost million, you absolutely need to get this right.
The new paper "Chinchilla Scaling: A replication attempt" by Epoch AI sets on on the ambitious goal of reproducing this.
But since the authors do not have infinite money, they decided to directly run their computations from DeepMind's own experiments! They took the figure from the last experiment (cf slide below), measured point positions, picked color codes, and ended up reconstructing the underlying data.
๐ฅ They then just fit the scaling laws proposed by the Chinchilla Authors, but arrived at wildly different results! They find that as a rough rule of thumb, you should use 20 training tokens for each parameter in your model, instead of the 70 obtained in the original paper. They also point out inconsistencies in the paper, and unrealistically narrow confidence intervals.
โก๏ธ This only contradicts the results from the last (out of 3) experiments in the Chinchilla paper. And the model trained at the end of the Chinchilla paper still seems properly scaled.
โ But it does show that a tiny bit more theoretical work can go a long way, especially given the huge financial costs that such an error can have!
posted
an
update
5 months ago
Post
2460
๐๐๐ฉ๐๐ซ ๐๐๐ฏ๐ข๐๐ฐ: ๐๐ก๐จ-๐ - ๐๐จ ๐ง๐จ๐ญ ๐ฎ๐ฌ๐ ๐๐ฅ๐ฅ ๐ญ๐จ๐ค๐๐ง๐ฌ ๐๐ช๐ฎ๐๐ฅ๐ฅ๐ฒ ๐ข๐ง ๐ฒ๐จ๐ฎ๐ซ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ! โ๏ธโ๏ธ
A new paper topping Daily papers questions a hidden assumption in LLM training:
๐ค ๐๐๐ค๐ช๐ก๐ ๐ฌ๐ ๐ง๐๐๐ก๐ก๐ฎ ๐ช๐จ๐ ๐๐ก๐ก ๐ฉ๐ค๐ ๐๐ฃ๐จ ๐๐ฆ๐ช๐๐ก๐ก๐ฎ ๐๐ฃ ๐ค๐ช๐ง ๐๐๐'๐จ ๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ?
Some tokens are more relevant than others, and some are mostly noise (just look up the history of ๐๐ฐ๐ญ๐ช๐ฅ๐๐ฐ๐ญ๐ฅ๐๐ข๐จ๐ช๐ฌ๐ข๐ณ๐ฑ).
So this paper introduces ๐ฆ๐ฒ๐น๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐ถ๐ป๐ด, which is actually really simple:
โก๏ธ A specific metric measures the relevance of each token. Then during training, only the top k% tokens for this relevance metric count in the loss calculation.
Authors test this method by training models on the difficult MATH dataset (only competition mathematics problems).
โก๏ธ Their technique seems like a new must-do in LLM training: Training is much faster and reaches an impressive performance!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
โ โฑ๏ธ Training is x5 to x10 faster to reach equivalent performance compared to standard language modeling.
โ ๐ช Their 1B model achieves close to GPT4 Chain-of-Thought performance on MATH!
โ ๐ Their 7B model match performance of the state-of-the-art DeepSeek for the same size, while trained on only 3% of tokens
๐๐๐๐ข๐ญ๐ข๐จ๐ง๐๐ฅ ๐ข๐ง๐ฌ๐ข๐ ๐ก๐ญ๐ฌ ๐ก
โ Datasets used for pre-training, even after pre-filtering, still contain a large proportion of noisy tokens ๐
โ Authors show that when you reduce loss on noisy tokens, you actually reduce accuracy (Figure 7). So Selective Language Modeling seems fundamental! โ
Find great reads in @akhaliq 's Daily Papers ๐ https://huggingface.co/papers
Paper added to my collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
A new paper topping Daily papers questions a hidden assumption in LLM training:
๐ค ๐๐๐ค๐ช๐ก๐ ๐ฌ๐ ๐ง๐๐๐ก๐ก๐ฎ ๐ช๐จ๐ ๐๐ก๐ก ๐ฉ๐ค๐ ๐๐ฃ๐จ ๐๐ฆ๐ช๐๐ก๐ก๐ฎ ๐๐ฃ ๐ค๐ช๐ง ๐๐๐'๐จ ๐ฉ๐ง๐๐๐ฃ๐๐ฃ๐ ?
Some tokens are more relevant than others, and some are mostly noise (just look up the history of ๐๐ฐ๐ญ๐ช๐ฅ๐๐ฐ๐ญ๐ฅ๐๐ข๐จ๐ช๐ฌ๐ข๐ณ๐ฑ).
So this paper introduces ๐ฆ๐ฒ๐น๐ฒ๐ฐ๐๐ถ๐๐ฒ ๐๐ฎ๐ป๐ด๐๐ฎ๐ด๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐น๐ถ๐ป๐ด, which is actually really simple:
โก๏ธ A specific metric measures the relevance of each token. Then during training, only the top k% tokens for this relevance metric count in the loss calculation.
Authors test this method by training models on the difficult MATH dataset (only competition mathematics problems).
โก๏ธ Their technique seems like a new must-do in LLM training: Training is much faster and reaches an impressive performance!
๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
โ โฑ๏ธ Training is x5 to x10 faster to reach equivalent performance compared to standard language modeling.
โ ๐ช Their 1B model achieves close to GPT4 Chain-of-Thought performance on MATH!
โ ๐ Their 7B model match performance of the state-of-the-art DeepSeek for the same size, while trained on only 3% of tokens
๐๐๐๐ข๐ญ๐ข๐จ๐ง๐๐ฅ ๐ข๐ง๐ฌ๐ข๐ ๐ก๐ญ๐ฌ ๐ก
โ Datasets used for pre-training, even after pre-filtering, still contain a large proportion of noisy tokens ๐
โ Authors show that when you reduce loss on noisy tokens, you actually reduce accuracy (Figure 7). So Selective Language Modeling seems fundamental! โ
Find great reads in @akhaliq 's Daily Papers ๐ https://huggingface.co/papers
Paper added to my collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
posted
an
update
5 months ago
Post
2160
๐ก๐ฒ๐ ๐ฆ๐ฝ๐ฎ๐ฐ๐ฒ: ๐ผ๐ ๐๐ง๐๐ซ๐๐ก ๐ฅ๐ก๐๐ฃ๐ฃ๐๐ง ๐บ๏ธ๐๏ธ Plan your next vacation in a few minutes!
I wanted to try out if a powerful LLM like Mixtral-8x7b had geographical reasoning capabilities.
So I built a small space that prompts the LLM to provide a JSON list of places based on a user input.
And the result was impressive! ๐คฏ
โ ๐๐ ๐๐ฒ๐ฒ๐บ๐ ๐น๐ถ๐ธ๐ฒ ๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น ๐ต๐ฎ๐ ๐ฎ ๐ด๐ฟ๐ฎ๐๐ฝ ๐ผ๐ณ ๐ด๐ฒ๐ผ๐ด๐ฟ๐ฎ๐ฝ๐ต๐ถ๐ฐ๐ฎ๐น ๐ฐ๐ผ๐ป๐ฐ๐ฒ๐ฝ๐๐ ๐น๐ถ๐ธ๐ฒ ๐ก๐ผ๐ฟ๐๐ต - ๐ฆ๐ผ๐๐๐ต, ๐ผ๐ฟ ๐๐ฝ๐ฎ๐๐ถ๐ฎ๐น ๐ฎ๐น๐ถ๐ด๐ป๐บ๐ฒ๐ป๐.๐งญ Not just describing these concepts, but really applying them in practice, for instance to successfully answer "give me 4 European cities that are aligned on the map". This is a ๐ป๐ถ๐ฐ๐ฒ ๐ฒ๐ ๐ฎ๐บ๐ฝ๐น๐ฒ ๐ผ๐ณ ๐ฎ๐ป ๐ฒ๐บ๐ฒ๐ฟ๐ด๐ฒ๐ป๐ ๐ฐ๐ฎ๐ฝ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐, since nothing in the LLM's training data should prepare it for this specific task.
Anyway, I added API calls and a nice visualization on top of the LLM, streaming output, caching for the answers and locations... and ta-da! โจ I got the ๐๐ ๐ง๐ฟ๐ฎ๐๐ฒ๐น ๐ฃ๐น๐ฎ๐ป๐ป๐ฒ๐ฟ.
๐๐ค๐ช ๐๐๐ฃ ๐๐๐จ๐๐ง๐๐๐ ๐๐ฉ ๐ฎ๐ค๐ช๐ง ๐ฉ๐ง๐๐ฅ, ๐๐ฃ๐ ๐๐ฉ ๐ฌ๐๐ก๐ก ๐๐ค๐ข๐ ๐ช๐ฅ ๐ฌ๐๐ฉ๐ ๐ฃ๐๐๐ ๐๐ฃ๐ ๐๐ค๐ฃ๐ซ๐๐ฃ๐๐๐ฃ๐ฉ ๐ก๐ค๐๐๐ฉ๐๐ค๐ฃ๐จ!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐ ๐ m-ric/ai-travel-planner
Thank you @freddyaboulton for the ๐๐๐๐๐๐_๐๐๐๐๐๐ component, and @clem , @pngwn , @abidlabs for your ideas and support!
I wanted to try out if a powerful LLM like Mixtral-8x7b had geographical reasoning capabilities.
So I built a small space that prompts the LLM to provide a JSON list of places based on a user input.
And the result was impressive! ๐คฏ
โ ๐๐ ๐๐ฒ๐ฒ๐บ๐ ๐น๐ถ๐ธ๐ฒ ๐ ๐ถ๐ ๐๐ฟ๐ฎ๐น ๐ต๐ฎ๐ ๐ฎ ๐ด๐ฟ๐ฎ๐๐ฝ ๐ผ๐ณ ๐ด๐ฒ๐ผ๐ด๐ฟ๐ฎ๐ฝ๐ต๐ถ๐ฐ๐ฎ๐น ๐ฐ๐ผ๐ป๐ฐ๐ฒ๐ฝ๐๐ ๐น๐ถ๐ธ๐ฒ ๐ก๐ผ๐ฟ๐๐ต - ๐ฆ๐ผ๐๐๐ต, ๐ผ๐ฟ ๐๐ฝ๐ฎ๐๐ถ๐ฎ๐น ๐ฎ๐น๐ถ๐ด๐ป๐บ๐ฒ๐ป๐.๐งญ Not just describing these concepts, but really applying them in practice, for instance to successfully answer "give me 4 European cities that are aligned on the map". This is a ๐ป๐ถ๐ฐ๐ฒ ๐ฒ๐ ๐ฎ๐บ๐ฝ๐น๐ฒ ๐ผ๐ณ ๐ฎ๐ป ๐ฒ๐บ๐ฒ๐ฟ๐ด๐ฒ๐ป๐ ๐ฐ๐ฎ๐ฝ๐ฎ๐ฏ๐ถ๐น๐ถ๐๐, since nothing in the LLM's training data should prepare it for this specific task.
Anyway, I added API calls and a nice visualization on top of the LLM, streaming output, caching for the answers and locations... and ta-da! โจ I got the ๐๐ ๐ง๐ฟ๐ฎ๐๐ฒ๐น ๐ฃ๐น๐ฎ๐ป๐ป๐ฒ๐ฟ.
๐๐ค๐ช ๐๐๐ฃ ๐๐๐จ๐๐ง๐๐๐ ๐๐ฉ ๐ฎ๐ค๐ช๐ง ๐ฉ๐ง๐๐ฅ, ๐๐ฃ๐ ๐๐ฉ ๐ฌ๐๐ก๐ก ๐๐ค๐ข๐ ๐ช๐ฅ ๐ฌ๐๐ฉ๐ ๐ฃ๐๐๐ ๐๐ฃ๐ ๐๐ค๐ฃ๐ซ๐๐ฃ๐๐๐ฃ๐ฉ ๐ก๐ค๐๐๐ฉ๐๐ค๐ฃ๐จ!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐ ๐ m-ric/ai-travel-planner
Thank you @freddyaboulton for the ๐๐๐๐๐๐_๐๐๐๐๐๐ component, and @clem , @pngwn , @abidlabs for your ideas and support!
- 1 reply
posted
an
update
6 months ago
Post
2067
[๐๐๐ฐ ๐๐๐ฉ๐๐ซ] ๐๐ฅ๐ฅ ๐ญ๐จ๐ค๐๐ง๐ฌ ๐ฌ๐ก๐จ๐ฎ๐ฅ๐ ๐ง๐จ๐ญ ๐ซ๐๐ช๐ฎ๐ข๐ซ๐ ๐ญ๐ก๐ ๐ฌ๐๐ฆ๐ ๐๐๐๐จ๐ซ๐ญ ๐ญ๐จ ๐๐จ๐ฆ๐ฉ๐ฎ๐ญ๐! โ ๐๐ข๐ฑ๐ญ๐ฎ๐ซ๐ ๐จ๐ ๐๐๐ฉ๐ญ๐ก๐ฌ ๐ซง๐
Google Researchers were unhappy with the way current decoding generally works: all tokens go through the same layers, thus requiring exactly the same effort to compute.
Whereas in reality, completing the answer to a difficult math problem for instance should be more computationally intense than completing the text of the Declaration of Independence: ๐ป๐ผ๐ ๐ฎ๐น๐น ๐๐ผ๐ธ๐ฒ๐ป๐ ๐ฎ๐ฟ๐ฒ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฒ๐พ๐๐ฎ๐น!
โก๏ธ ๐ง๐ต๐ฒ๐ ๐ต๐ฎ๐ฑ ๐๐ต๐ถ๐ ๐ด๐ฒ๐ป๐ถ๐๐ ๐ถ๐ฑ๐ฒ๐ฎ: ๐ก ๐ต๐ฎ๐๐ถ๐ป๐ด ๐ฎ ๐๐ผ๐ธ๐ฒ๐ป ๐ด๐ผ ๐๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฎ ๐ฏ๐น๐ผ๐ฐ๐ธ ๐๐ต๐ผ๐๐น๐ฑ ๐ฏ๐ฒ ๐ผ๐ฝ๐๐ถ๐ผ๐ป๐ฎ๐น. The token can go through the block (thus undergoing expensive self-attention computation) or avoid it through a skip connection.
The routing decision is taken on the block level: each block selects from the total sequence the top-k tokens that will go through it, and the others tokens will skip it. ๐๐ฉ๐ช๐ด ๐ข๐ญ๐ญ๐ฐ๐ธ๐ด ๐ต๐ฐ ๐ค๐ฉ๐ฐ๐ฐ๐ด๐ฆ ๐ต๐ฉ๐ฆ ๐ฆ๐น๐ข๐ค๐ต ๐๐๐ฅ๐๐๐๐ฉ๐ฎ ๐ฐ๐ง ๐ข ๐ฃ๐ญ๐ฐ๐ค๐ฌ, ๐ช.๐ฆ. ๐ต๐ฉ๐ฆ ๐ฑ๐ณ๐ฐ๐ฑ๐ฐ๐ณ๐ต๐ช๐ฐ๐ฏ ๐ฐ๐ง ๐ต๐ฐ๐ฌ๐ฆ๐ฏ๐ด ๐ต๐ฉ๐ข๐ต ๐จ๐ฐ ๐ต๐ฉ๐ณ๐ฐ๐ถ๐จ๐ฉ ๐ช๐ต, ๐ธ๐ฉ๐ช๐ค๐ฉ ๐ฅ๐ช๐ณ๐ฆ๐ค๐ต๐ญ๐บ ๐ช๐ฏ๐ง๐ญ๐ถ๐ฆ๐ฏ๐ค๐ฆ๐ด ๐ต๐ฉ๐ฆ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ข๐ต๐ช๐ฐ๐ฏ๐ข๐ญ ๐ช๐ฏ๐ต๐ฆ๐ฏ๐ด๐ช๐ต๐บ ๐ฐ๐ง ๐ต๐ฉ๐ฆ ๐ง๐ฐ๐ณ๐ธ๐ข๐ณ๐ฅ ๐ฑ๐ข๐ด๐ด.
This yields Mixture-of-Depths (MoD), with spectacular results.
โจ ๐ฅ๐ฒ๐๐๐น๐๐:
๐๏ธ ๐๐ฎ๐ฝ๐ฎ๐ฐ๐ถ๐๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฒ ๐๐๐ป๐ฒ๐ฑ ๐ฎ๐น๐น ๐๐ต๐ฒ ๐๐ฎ๐ ๐ฑ๐ผ๐๐ป ๐๐ผ ๐ญ๐ฎ.๐ฑ% for every second block: thus 87.5% of tokens just skip the block!
๐ For the same training time and performance, >๐ฒ๐ฌ% ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐๐ฝ๐ฒ๐ฒ๐ฑ!
๐ค ๐๐ฎ๐ป ๐ฏ๐ฒ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฒ๐ฑ ๐๐ถ๐๐ต ๐ ๐ถ๐ ๐๐๐ฟ๐ฒ-๐ผ๐ณ-๐๐ ๐ฝ๐ฒ๐ฟ๐๐ for further improvements.
๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ ๐ต๐ฒ๐ฟ๐ฒ ๐ Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (2404.02258)
๐ I added it to my paper collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
Google Researchers were unhappy with the way current decoding generally works: all tokens go through the same layers, thus requiring exactly the same effort to compute.
Whereas in reality, completing the answer to a difficult math problem for instance should be more computationally intense than completing the text of the Declaration of Independence: ๐ป๐ผ๐ ๐ฎ๐น๐น ๐๐ผ๐ธ๐ฒ๐ป๐ ๐ฎ๐ฟ๐ฒ ๐ฐ๐ฟ๐ฒ๐ฎ๐๐ฒ๐ฑ ๐ฒ๐พ๐๐ฎ๐น!
โก๏ธ ๐ง๐ต๐ฒ๐ ๐ต๐ฎ๐ฑ ๐๐ต๐ถ๐ ๐ด๐ฒ๐ป๐ถ๐๐ ๐ถ๐ฑ๐ฒ๐ฎ: ๐ก ๐ต๐ฎ๐๐ถ๐ป๐ด ๐ฎ ๐๐ผ๐ธ๐ฒ๐ป ๐ด๐ผ ๐๐ต๐ฟ๐ผ๐๐ด๐ต ๐ฎ ๐ฏ๐น๐ผ๐ฐ๐ธ ๐๐ต๐ผ๐๐น๐ฑ ๐ฏ๐ฒ ๐ผ๐ฝ๐๐ถ๐ผ๐ป๐ฎ๐น. The token can go through the block (thus undergoing expensive self-attention computation) or avoid it through a skip connection.
The routing decision is taken on the block level: each block selects from the total sequence the top-k tokens that will go through it, and the others tokens will skip it. ๐๐ฉ๐ช๐ด ๐ข๐ญ๐ญ๐ฐ๐ธ๐ด ๐ต๐ฐ ๐ค๐ฉ๐ฐ๐ฐ๐ด๐ฆ ๐ต๐ฉ๐ฆ ๐ฆ๐น๐ข๐ค๐ต ๐๐๐ฅ๐๐๐๐ฉ๐ฎ ๐ฐ๐ง ๐ข ๐ฃ๐ญ๐ฐ๐ค๐ฌ, ๐ช.๐ฆ. ๐ต๐ฉ๐ฆ ๐ฑ๐ณ๐ฐ๐ฑ๐ฐ๐ณ๐ต๐ช๐ฐ๐ฏ ๐ฐ๐ง ๐ต๐ฐ๐ฌ๐ฆ๐ฏ๐ด ๐ต๐ฉ๐ข๐ต ๐จ๐ฐ ๐ต๐ฉ๐ณ๐ฐ๐ถ๐จ๐ฉ ๐ช๐ต, ๐ธ๐ฉ๐ช๐ค๐ฉ ๐ฅ๐ช๐ณ๐ฆ๐ค๐ต๐ญ๐บ ๐ช๐ฏ๐ง๐ญ๐ถ๐ฆ๐ฏ๐ค๐ฆ๐ด ๐ต๐ฉ๐ฆ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ข๐ต๐ช๐ฐ๐ฏ๐ข๐ญ ๐ช๐ฏ๐ต๐ฆ๐ฏ๐ด๐ช๐ต๐บ ๐ฐ๐ง ๐ต๐ฉ๐ฆ ๐ง๐ฐ๐ณ๐ธ๐ข๐ณ๐ฅ ๐ฑ๐ข๐ด๐ด.
This yields Mixture-of-Depths (MoD), with spectacular results.
โจ ๐ฅ๐ฒ๐๐๐น๐๐:
๐๏ธ ๐๐ฎ๐ฝ๐ฎ๐ฐ๐ถ๐๐ ๐ฐ๐ฎ๐ป ๐ฏ๐ฒ ๐๐๐ป๐ฒ๐ฑ ๐ฎ๐น๐น ๐๐ต๐ฒ ๐๐ฎ๐ ๐ฑ๐ผ๐๐ป ๐๐ผ ๐ญ๐ฎ.๐ฑ% for every second block: thus 87.5% of tokens just skip the block!
๐ For the same training time and performance, >๐ฒ๐ฌ% ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ ๐๐ฝ๐ฒ๐ฒ๐ฑ!
๐ค ๐๐ฎ๐ป ๐ฏ๐ฒ ๐ฐ๐ผ๐บ๐ฏ๐ถ๐ป๐ฒ๐ฑ ๐๐ถ๐๐ต ๐ ๐ถ๐ ๐๐๐ฟ๐ฒ-๐ผ๐ณ-๐๐ ๐ฝ๐ฒ๐ฟ๐๐ for further improvements.
๐ ๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ ๐ต๐ฒ๐ฟ๐ฒ ๐ Mixture-of-Depths: Dynamically allocating compute in transformer-based language models (2404.02258)
๐ I added it to my paper collection ๐ m-ric/spinning-up-in-llms-659e698f9dd5a71bd3f579a7
- 1 reply
posted
an
update
6 months ago
Post
1875
๐๐๐๐, ๐ญ๐ก๐ ๐ฒ๐๐๐ซ ๐จ๐ ๐๐ ๐๐ง๐ญ ๐ฐ๐จ๐ซ๐ค๐๐ฅ๐จ๐ฐ๐ฌ ๐ง๐ฆพ๐ค
I've just watched Andrew Ng's talk at Sequoia last week.
If you're interested in Agents, you should really watch it!
๐ช๐ต๐ ๐๐๐ฒ ๐ฎ๐ด๐ฒ๐ป๐ ๐๐ผ๐ฟ๐ธ๐ณ๐น๐ผ๐๐?
The current LLM task solving workflow is not very intuitive:
We ask it โwrite an essay all in one shot, without ever using backspace.โ
Why not allow the LLM a more similar process to what we would do?
- โWrite an essay outlineโ
- โDo you need wen research?โ
- โWrite a first draftโ
- โConsider improvementsโ
โฆ
This is called an Agentic workflow. Existing ones bring a huge performance boost. With HumanEval: GPT-4 zero-shot gets 67% score, agentic with either one of tool use or reflection goes over 90%, and the combination of the two scores even higher!
๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐ฟ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐ฑ๐ฒ๐๐ถ๐ด๐ป ๐ฝ๐ฎ๐๐๐ฒ๐ฟ๐ป๐
On the following two points, the tech is robust:
โ๏ธ ๐ฅ๐ฒ๐ณ๐น๐ฒ๐ ๐ถ๐ผ๐ป: For instance: add a critic step after the writing step
๐ ๏ธ ๐ง๐ผ๐ผ๐น ๐๐๐ฒ: extends the capabilities of the LLM by allowing it to call tools, like search or calculator
The next two will be needed to go further, but the tech for them is more emerging and not reliable yet:
๐บ๏ธ ๐ฃ๐น๐ฎ๐ป๐ป๐ถ๐ป๐ด forward to decompose task into subtasks. This allows great behaviours like an AI Agent re-routing after a failure
๐ ๐ ๐๐น๐๐ถ-๐ฎ๐ด๐ฒ๐ป๐ ๐ฐ๐ผ๐น๐น๐ฎ๐ฏ๐ผ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Program a flock of agents with tasks.
Improving the two above points will unlock huge performance boosts!
Andrew NG says Research agents are already part of his workflow!
๐๐น๐ผ๐๐ถ๐ป๐ด ๐๐ต๐ผ๐๐ด๐ต๐๐
Andrew speculates that through agentic workflows, maybe generating many tokens fast from a small LLM will give better results than slower throughput from a powerful LLM like GPT-5.
๐ฌ Watch the talk here ๐ https://www.youtube.com/watch?v=sal78ACtGTc
๐ I've added his recommended reads to m-ric/agents-65ba776fbd9e29f771c07d4e
I've just watched Andrew Ng's talk at Sequoia last week.
If you're interested in Agents, you should really watch it!
๐ช๐ต๐ ๐๐๐ฒ ๐ฎ๐ด๐ฒ๐ป๐ ๐๐ผ๐ฟ๐ธ๐ณ๐น๐ผ๐๐?
The current LLM task solving workflow is not very intuitive:
We ask it โwrite an essay all in one shot, without ever using backspace.โ
Why not allow the LLM a more similar process to what we would do?
- โWrite an essay outlineโ
- โDo you need wen research?โ
- โWrite a first draftโ
- โConsider improvementsโ
โฆ
This is called an Agentic workflow. Existing ones bring a huge performance boost. With HumanEval: GPT-4 zero-shot gets 67% score, agentic with either one of tool use or reflection goes over 90%, and the combination of the two scores even higher!
๐๐ด๐ฒ๐ป๐๐ถ๐ฐ ๐ฟ๐ฒ๐ฎ๐๐ผ๐ป๐ถ๐ป๐ด ๐ฑ๐ฒ๐๐ถ๐ด๐ป ๐ฝ๐ฎ๐๐๐ฒ๐ฟ๐ป๐
On the following two points, the tech is robust:
โ๏ธ ๐ฅ๐ฒ๐ณ๐น๐ฒ๐ ๐ถ๐ผ๐ป: For instance: add a critic step after the writing step
๐ ๏ธ ๐ง๐ผ๐ผ๐น ๐๐๐ฒ: extends the capabilities of the LLM by allowing it to call tools, like search or calculator
The next two will be needed to go further, but the tech for them is more emerging and not reliable yet:
๐บ๏ธ ๐ฃ๐น๐ฎ๐ป๐ป๐ถ๐ป๐ด forward to decompose task into subtasks. This allows great behaviours like an AI Agent re-routing after a failure
๐ ๐ ๐๐น๐๐ถ-๐ฎ๐ด๐ฒ๐ป๐ ๐ฐ๐ผ๐น๐น๐ฎ๐ฏ๐ผ๐ฟ๐ฎ๐๐ถ๐ผ๐ป: Program a flock of agents with tasks.
Improving the two above points will unlock huge performance boosts!
Andrew NG says Research agents are already part of his workflow!
๐๐น๐ผ๐๐ถ๐ป๐ด ๐๐ต๐ผ๐๐ด๐ต๐๐
Andrew speculates that through agentic workflows, maybe generating many tokens fast from a small LLM will give better results than slower throughput from a powerful LLM like GPT-5.
๐ฌ Watch the talk here ๐ https://www.youtube.com/watch?v=sal78ACtGTc
๐ I've added his recommended reads to m-ric/agents-65ba776fbd9e29f771c07d4e
- 1 reply
posted
an
update
6 months ago
Post
1793
๐๐ก๐ ๐ซ๐๐ญ๐ฎ๐ซ๐ง ๐จ๐ ๐ญ๐ก๐ ๐๐๐๐ฌ โ ๐๐๐ฐ ๐๐๐ฆ๐๐-๐๐๐ฌ๐๐ ๐๐ซ๐๐ก๐ข๐ญ๐๐๐ญ๐ฎ๐ซ๐ "๐๐๐ฆ๐๐"
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
Since the release of BERT by Google in 2019, Transformers architecture have taken over machine learning thanks to their ๐ฎ๐๐๐ฒ๐ป๐๐ถ๐ผ๐ป ๐บ๐ฒ๐ฐ๐ต๐ฎ๐ป๐ถ๐๐บ, that gives them the ability to focus on important points of the input. But ๐๐ฉ๐ฉ๐๐ฃ๐ฉ๐๐ค๐ฃ ๐๐ค๐ข๐ฅ๐ช๐ฉ๐๐ฉ๐๐ค๐ฃ ๐๐จ ๐ฆ๐ช๐๐๐ง๐๐ฉ๐๐ ๐๐ฃ ๐ฉ๐๐ ๐๐ฃ๐ฅ๐ช๐ฉ ๐ก๐๐ฃ๐๐ฉ๐.
๐ซ The Mamba paper, published in December 2023, announced the return of the RNNs: it has no attention, but integrates a selection mechanism, which should be able to reproduce the โfocusโ ability of attention, in an architecture for which the compute requirements ๐ด๐ฟ๐ผ๐ ๐ผ๐ป๐น๐ ๐น๐ถ๐ป๐ฒ๐ฎ๐ฟ๐น๐ ๐ถ๐ป ๐ถ๐ป๐ฝ๐๐ ๐น๐ฒ๐ป๐ด๐๐ต!
๐ค Would this work? We had yet to see a large Mamba model recovering the performance of Attention-based Transformers.
๐ฅ But now it's done! A (Mamba + Transformers) hybrid just beat Transformers!
The AI21 Labs team just released Jamba.
They insert a few Transformer layers to inject some attention in a big pile of Mamba layers, thus getting the best of both worlds.
๐๐;๐ฟ๐:
๐๏ธ ๐ก๐ฒ๐ ๐ ๐ผ๐ ๐ฎ๐ฟ๐ฐ๐ต๐ถ๐๐ฒ๐ฐ๐๐๐ฟ๐ฒ: 4 Jamba blocks, each of these being 7 Mamba layers for 1 Transformer.
๐๏ธ ๐ฑ๐ฎ๐ ๐ฝ๐ฎ๐ฟ๐ฎ๐บ๐ฒ๐๐ฒ๐ฟ๐, ๐ญ๐ฎ๐ ๐ฎ๐ฐ๐๐ถ๐๐ฒ ๐ฎ๐ ๐ถ๐ป๐ณ๐ฒ๐ฟ๐ฒ๐ป๐ฐ๐ฒ: This reduction is enabled by Mixture of Experts, and similar to Mixtral (47B parameters - 13B active).
๐๏ธ ๐ฆ๐ฝ๐ฒ๐ฒ๐ฑ: ๐ ๐ฏ ๐๐ต๐ฟ๐ผ๐๐ด๐ต๐ฝ๐๐. Jamba is much faster than similar-sized Transformer models on long contexts.
๐ ๐๐ผ๐ป๐๐ฒ๐ ๐ ๐น๐ฒ๐ป๐ด๐๐ต: ๐ญ๐ฐ๐ฌ๐ ๐๐ผ๐ธ๐ฒ๐ป๐ on a single 80GB A100!
๐ช ๐ฃ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ: ๐๐๐ฎ๐๐ฒ-๐ผ๐ณ-๐๐ต๐ฒ-๐ฎ๐ฟ๐ ๐ณ๐ผ๐ฟ ๐๐ต๐ถ๐ ๐๐ถ๐๐ฒ. The small injection of attention seems sufficient since Jamba beats the open-source reference Mixtral-8x7B on many benchmarks!
Try it here ๐ ai21labs/Jamba-v0.1
posted
an
update
6 months ago
Post
1697
๐๐ผ๐ ๐ฑ๐ผ๐ฒ๐ ๐ฏ๐ฒ๐ฎ๐บ ๐๐ฒ๐ฎ๐ฟ๐ฐ๐ต ๐ฑ๐ฒ๐ฐ๐ผ๐ฑ๐ถ๐ป๐ด ๐๐ผ๐ฟ๐ธ? โก๏ธ ๐๐๐ฌ ๐ซ๐๐จ๐ช๐๐ก๐๐ฏ๐๐ฉ๐๐ค๐ฃ ๐ฉ๐ค๐ค๐ก! ๐
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
In Decoder-type LLMs like GPT4 or Mistral-Large, the output is generated one token (=word part) at a time. That's why they're nicknamed "stochastic parrots": the "thinking" process only happens one step at a time, so it can seem really myopic.
๐๐จ ๐ก๐จ๐ฐ ๐ข๐ฌ ๐ญ๐ก๐ ๐ง๐๐ฑ๐ญ ๐ญ๐จ๐ค๐๐ง ๐ฌ๐๐ฅ๐๐๐ญ๐๐?
๐ Given its input sentence like "๐๐ฉ๐ข๐ต ๐ช๐ด ๐ต๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ? ๐๐ฉ๐ฆ 7๐ต๐ฉ ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ", the Decoder LLM generates, for each token in its vocabulary, a score that represents this token's probability of coming next.
For instance: "๐๐จ" gets score 0.56, and "๐๐๐ฃ" gets score 0.35.
๐ค ๐๐ซ๐๐๐๐ฒ ๐๐๐๐จ๐๐ข๐ง๐ is the naive option where you simply take the next most probable token at each step. But this creates paths that maximize very short-term rewards, thus may overlook better paths for the long term (like this time when you played FIFA all evening and arrived unprepared to your school exam on the next day).
In our example, the next highest score token might be "๐๐จ", but this will strongly bias the LLM towards giving an hasty response. On the opposite, starting with "๐๐๐ฃ" could have been completed with "๐ฃ๐ฆ ๐ฐ๐ฃ๐ต๐ข๐ช๐ฏ๐ฆ๐ฅ ๐ง๐ณ๐ฐ๐ฎ ๐ค๐ฐ๐ฎ๐ฑ๐ถ๐ต๐ช๐ฏ๐จ ๐ฑ๐ณ๐ฆ๐ท๐ช๐ฐ๐ถ๐ด ๐๐ช๐ฃ๐ฐ๐ฏ๐ข๐ค๐ค๐ช ๐ฏ๐ถ๐ฎ๐ฃ๐ฆ๐ณ๐ด ๐ง๐ช๐ณ๐ด๐ต", which steers the LLM towards a correct reasoning!
๐บ๏ธ ๐๐๐๐ฆ ๐ฌ๐๐๐ซ๐๐ก improves on greedy decoding by generating at each step several paths - called beams - instead of one. This allows the generation to explore a much larger space, thus find better completions. In our example, both the "๐๐จ" and the "๐๐๐ฃ" completion could be tested. โ
๐ I've created a tool to let you visualize it, thank you @joaogante for your great help!
๐๐ง๐ฎ ๐๐ฉ ๐๐๐ง๐: m-ric/beam_search_visualizer
posted
an
update
6 months ago
Post
2033
๐จ๐๐ถ๐ป๐ด ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐งโโ๏ธ ๐ณ๐ผ๐ฟ ๐ฎ๐ป ๐ฎ๐๐๐ผ๐บ๐ฎ๐๐ฒ๐ฑ ๐ฎ๐ป๐ฑ ๐๐ฒ๐ฟ๐๐ฎ๐๐ถ๐น๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป
Evaluating LLM outputs is often hard, since many tasks require open-ended answers for which no deterministic metrics work: for instance, when asking a model to summarize a text, there could be hundreds of correct ways to do it. The most versatile way to grade these outputs is then human evaluation, but it is very time-consuming, thus costly.
๐ค Then ๐๐ต๐ ๐ป๐ผ๐ ๐ฎ๐๐ธ ๐ฎ๐ป๐ผ๐๐ต๐ฒ๐ฟ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป, by providing it relevant rating criteria? ๐ This is the idea behind LLM-as-a-judge.
โ๏ธ To implement a LLM judge correctly, you need a few tricks.
โ So ๐'๐๐ฒ ๐ท๐๐๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐ฎ ๐ป๐ฒ๐ ๐ป๐ผ๐๐ฒ๐ฏ๐ผ๐ผ๐ธ ๐๐ต๐ผ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ผ ๐ถ๐บ๐ฝ๐น๐ฒ๐บ๐ฒ๐ป๐ ๐ถ๐ ๐ฝ๐ฟ๐ผ๐ฝ๐ฒ๐ฟ๐น๐ ๐ถ๐ป ๐ผ๐๐ฟ ๐๐๐ด๐ด๐ถ๐ป๐ด ๐๐ฎ๐ฐ๐ฒ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ! (you can run it instantly in Google Colab)
โก๏ธ ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: https://huggingface.co/learn/cookbook/llm_judge
The Cookbook is a great collection of notebooks demonstrating recipes (thus the "cookbook") for common LLM usages. I recommend you to go take a look!
โก๏ธ ๐๐น๐น ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ๐: https://huggingface.co/learn/cookbook/index
Thank you @MariaK for your support!
Evaluating LLM outputs is often hard, since many tasks require open-ended answers for which no deterministic metrics work: for instance, when asking a model to summarize a text, there could be hundreds of correct ways to do it. The most versatile way to grade these outputs is then human evaluation, but it is very time-consuming, thus costly.
๐ค Then ๐๐ต๐ ๐ป๐ผ๐ ๐ฎ๐๐ธ ๐ฎ๐ป๐ผ๐๐ต๐ฒ๐ฟ ๐๐๐ ๐๐ผ ๐ฑ๐ผ ๐๐ต๐ฒ ๐ฒ๐๐ฎ๐น๐๐ฎ๐๐ถ๐ผ๐ป, by providing it relevant rating criteria? ๐ This is the idea behind LLM-as-a-judge.
โ๏ธ To implement a LLM judge correctly, you need a few tricks.
โ So ๐'๐๐ฒ ๐ท๐๐๐ ๐ฝ๐๐ฏ๐น๐ถ๐๐ต๐ฒ๐ฑ ๐ฎ ๐ป๐ฒ๐ ๐ป๐ผ๐๐ฒ๐ฏ๐ผ๐ผ๐ธ ๐๐ต๐ผ๐๐ถ๐ป๐ด ๐ต๐ผ๐ ๐๐ผ ๐ถ๐บ๐ฝ๐น๐ฒ๐บ๐ฒ๐ป๐ ๐ถ๐ ๐ฝ๐ฟ๐ผ๐ฝ๐ฒ๐ฟ๐น๐ ๐ถ๐ป ๐ผ๐๐ฟ ๐๐๐ด๐ด๐ถ๐ป๐ด ๐๐ฎ๐ฐ๐ฒ ๐๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ! (you can run it instantly in Google Colab)
โก๏ธ ๐๐๐ -๐ฎ๐-๐ฎ-๐ท๐๐ฑ๐ด๐ฒ ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ: https://huggingface.co/learn/cookbook/llm_judge
The Cookbook is a great collection of notebooks demonstrating recipes (thus the "cookbook") for common LLM usages. I recommend you to go take a look!
โก๏ธ ๐๐น๐น ๐ฐ๐ผ๐ผ๐ธ๐ฏ๐ผ๐ผ๐ธ๐: https://huggingface.co/learn/cookbook/index
Thank you @MariaK for your support!
- 2 replies
posted
an
update
6 months ago
Post
Interesting paper: ๐๐๐๐จ๐ซ๐: ๐ญ๐ซ๐๐ข๐ง ๐๐ ๐ฆ๐จ๐๐๐ฅ๐ฌ ๐จ๐ง ๐๐จ๐ง๐ฌ๐ฎ๐ฆ๐๐ซ-๐ ๐ซ๐๐๐ ๐๐๐๐ฌ ๐ช
It's now possible to ๐๐ช๐ก๐ก๐ฎ ๐ฅ๐ง๐-๐ฉ๐ง๐๐๐ฃ a 7B model on a consumer-grade GPU of 24Gb RAM, without any performance loss!
The memory usage of training models has always been an acute issue. For instance full pre-training of a 7B model used to eat ~50Gb of RAM!
The common workarounds to reduce memory load are:
- separate models on multiple GPUs ("sharding")
- quantize models: encode weights on fewer bits
Another technique is to ๐ฅ๐ง๐ค๐๐๐๐ฉ ๐ฉ๐๐ ๐ฌ๐๐๐๐๐ฉ ๐ข๐๐ฉ๐ง๐๐ญ ๐ฉ๐ค ๐ก๐ค๐ฌ๐๐ง-๐ง๐๐ฃ๐ ๐จ๐ฅ๐๐๐๐จ, (since sometimes the weights do not really vary on all dimensions): this can save a lot of space!
This low-rank projection can be done on adapters to preserve the original weights (go check out LoRA), but it still generally hurts the performance too much for pre-training.
โก๏ธ Enter the authors of ๐๐ข๐๐ฐ๐ณ๐ฆ: ๐๐ฆ๐ฎ๐ฐ๐ณ๐บ-๐๐ง๐ง๐ช๐ค๐ช๐ฆ๐ฏ๐ต ๐๐๐ ๐๐ณ๐ข๐ช๐ฏ๐ช๐ฏ๐จ ๐ฃ๐บ ๐๐ณ๐ข๐ฅ๐ช๐ฆ๐ฏ๐ต ๐๐ฐ๐ธ-๐๐ข๐ฏ๐ฌ ๐๐ณ๐ฐ๐ซ๐ฆ๐ค๐ต๐ช๐ฐ๐ฏ. They gather (and prove) interesting insights:
โ The weight matrix does not reliably converge to lower ranks during training.
โ But the gradient matrix does!
Based on these insights, ๐๐ต๐ฒ๐ ๐ฏ๐๐ถ๐น๐ฑ ๐๐ฎ๐๐ผ๐ฟ๐ฒ, that projects the gradient to lower ranks.
๐บ๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ถ๐ฑ๐ฒ๐ฎ: to leave the optimization free to explore more space, they periodically re-build the low-rank projection throughout the training (a nice illustration is in the paper).
๐ค This method can even be combined with previous ones such as 8-bit Adam (quantizing the optimizer states to 8-bit).
โก๏ธ ๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Of course, huge reduction in memory footprint allowing the training on consumer-grade GPU (cf figure).
๐ช No reduction in performance: this scales well up to 7B parameters (and was independently confirmed since) โ this is essential, it confirms that the method is viable!
Read the full paper here: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)
It's now possible to ๐๐ช๐ก๐ก๐ฎ ๐ฅ๐ง๐-๐ฉ๐ง๐๐๐ฃ a 7B model on a consumer-grade GPU of 24Gb RAM, without any performance loss!
The memory usage of training models has always been an acute issue. For instance full pre-training of a 7B model used to eat ~50Gb of RAM!
The common workarounds to reduce memory load are:
- separate models on multiple GPUs ("sharding")
- quantize models: encode weights on fewer bits
Another technique is to ๐ฅ๐ง๐ค๐๐๐๐ฉ ๐ฉ๐๐ ๐ฌ๐๐๐๐๐ฉ ๐ข๐๐ฉ๐ง๐๐ญ ๐ฉ๐ค ๐ก๐ค๐ฌ๐๐ง-๐ง๐๐ฃ๐ ๐จ๐ฅ๐๐๐๐จ, (since sometimes the weights do not really vary on all dimensions): this can save a lot of space!
This low-rank projection can be done on adapters to preserve the original weights (go check out LoRA), but it still generally hurts the performance too much for pre-training.
โก๏ธ Enter the authors of ๐๐ข๐๐ฐ๐ณ๐ฆ: ๐๐ฆ๐ฎ๐ฐ๐ณ๐บ-๐๐ง๐ง๐ช๐ค๐ช๐ฆ๐ฏ๐ต ๐๐๐ ๐๐ณ๐ข๐ช๐ฏ๐ช๐ฏ๐จ ๐ฃ๐บ ๐๐ณ๐ข๐ฅ๐ช๐ฆ๐ฏ๐ต ๐๐ฐ๐ธ-๐๐ข๐ฏ๐ฌ ๐๐ณ๐ฐ๐ซ๐ฆ๐ค๐ต๐ช๐ฐ๐ฏ. They gather (and prove) interesting insights:
โ The weight matrix does not reliably converge to lower ranks during training.
โ But the gradient matrix does!
Based on these insights, ๐๐ต๐ฒ๐ ๐ฏ๐๐ถ๐น๐ฑ ๐๐ฎ๐๐ผ๐ฟ๐ฒ, that projects the gradient to lower ranks.
๐บ๏ธ ๐๐ฟ๐ฒ๐ฎ๐ ๐ถ๐ฑ๐ฒ๐ฎ: to leave the optimization free to explore more space, they periodically re-build the low-rank projection throughout the training (a nice illustration is in the paper).
๐ค This method can even be combined with previous ones such as 8-bit Adam (quantizing the optimizer states to 8-bit).
โก๏ธ ๐๐๐ฌ๐ฎ๐ฅ๐ญ๐ฌ:
๐ Of course, huge reduction in memory footprint allowing the training on consumer-grade GPU (cf figure).
๐ช No reduction in performance: this scales well up to 7B parameters (and was independently confirmed since) โ this is essential, it confirms that the method is viable!
Read the full paper here: GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection (2403.03507)
posted
an
update
7 months ago
Post
๐๐ If you're building RAG applications, you should check this out:
โ๏ธ I've built a new space to let you visualize the chunks you get with different text splitting methods!
โก๏ธ Visualize your chunks here:
m-ric/chunk_visualizer
โ๏ธ I've built a new space to let you visualize the chunks you get with different text splitting methods!
โก๏ธ Visualize your chunks here:
m-ric/chunk_visualizer
- 2 replies