๐ฎ๐น๐ฏ๐ต๐ง๐ท Generating multilingual instruction datasets with Magpie ๐ฆโโฌ
โข
18
Stefano Fiorucci PRO
anakin87
AI & ML interests
Contributing to Haystack, the LLM Framework ๐๏ธ. NLP / LLMs.
Articles
Organizations
anakin87's activity
reacted to
davanstrien's
post with โค๏ธ
1 day ago
Post
1071
huggingface.co/DIBT is dead! Long live https://huggingface.co/data-is-better-together!
We're working on some very cool projects so we're doing a bit of tidying of the Data is Better Together Hub org ๐ค
replied to
their
post
about 1 month ago
๐ก ๐๐๐ ๐ฉ๐ข๐ ๐ฐ๐ข๐ญ๐ก ๐ฌ๐ฒ๐ฌ๐ญ๐๐ฆ ๐ฆ๐๐ฌ๐ฌ๐๐ ๐
I had another idea: use the system message to steer generation towards a specific language.
The system message should be in the target language, like:
"You are an artificial intelligence that answers users' questions in TARGET_LANGUAGE in a useful and detailed way. The user asks complex questions in TARGET_LANGUAGE."
It is a simple approach, but it might work...
It turns out the authors had a similar idea, which they included in the latest revision of their paper. ๐
๐ช Resources
Magpie paper and repository: https://huggingface.co/papers/2406.08464 https://github.com/magpie-align/magpie
Magpie demo by @davanstrien : https://huggingface.co/spaces/davanstrien/magpie
Magpie Ollama Datagen by @mrm8488 : https://github.com/mrm8488/magpie-ollama-datagen
magpie-ultra dataset - massive dataset built with Magpie by Argilla: https://huggingface.co/datasets/argilla/magpie-ultra-v0.1
โ๏ธ distilabel framework - framework for synthetic data generation and AI feedback at scale: https://distilabel.argilla.io/latest/
posted
an
update
about 1 month ago
Post
1033
Ok, you're finally convinced that synthetic data works... โ๏ธ
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
๐๐จ๐ฐ ๐ฒ๐จ๐ฎ ๐ฐ๐๐ง๐ญ ๐ญ๐จ ๐ ๐๐ง๐๐ซ๐๐ญ๐ ๐๐ง ๐ข๐ง๐ฌ๐ญ๐ซ๐ฎ๐๐ญ๐ข๐จ๐ง ๐๐๐ญ๐๐ฌ๐๐ญ ๐๐จ๐ซ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ข๐ง ๐ ๐ฅ๐๐ง๐ ๐ฎ๐๐ ๐ ๐จ๐ญ๐ก๐๐ซ ๐ญ๐ก๐๐ง ๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก.
But how do you get started?
I explore how to do this with Magpie in my new article
https://huggingface.co/blog/anakin87/multilingual-magpie
---
๐ฆโโฌ ๐๐ก๐๐ญ ๐ข๐ฌ ๐๐๐ ๐ฉ๐ข๐?
It's a recent technique for creating synthetic instruction datasets.
Magpie is based on a simple but ingenious idea ๐
if you prompt an instruction-tuned model with a pre-query template, you can make it generate a plausible user query/instruction
Here's an example:
model: Llama-3-8B-Instruct
pre-query template: "<|begin_of_text|><|start_header_id|>user<|end_header_id|>"
generated user instruction: "What are some of the responsibilities of a commercial pilot?"
You can then feed this instruction back into the same model to get the assistant response.
By repeating this process, it's possible to generate large synthetic datasets with relatively little effort.
๐ช The authors demonstrate that using these datasets for Supervised Fine Tuning (SFT) can yield strong performance, even competitive with the original instruct model.
๐ง๐๐๐ง๐๐ซ๐๐ญ๐ข๐ง๐ ๐ง๐จ๐ง-๐๐ง๐ ๐ฅ๐ข๐ฌ๐ก ๐๐๐ญ๐
Most Language Models are primarily trained on English texts, so they tend to produce data in English.
How can we overcome this?
Earlier approaches were complex or costly.
Then @mrm8488 found a simple solution: add the target language to the pre-query template.
For Spanish, the template becomes "<|begin_of_text|><|start_header_id|>user<|end_header_id|>spanish:".
This method works for Spanish and German!
โ Unfortunately, it does not work well for other languages (๐ฎ๐น, ๐ณ๐ฑ, ...)
๐
-
- 1 reply
posted
an
update
about 2 months ago
Post
1693
๐ต๐ป ๐๐ ๐๐ง๐ญ๐ข๐ ๐๐๐ ๐ฐ๐ข๐ญ๐ก ๐ฆ ๐๐ฅ๐๐ฆ๐ 3.2
I was excited to explore Llama 3.2, but as a simple ๐ช๐บ EU guy, I don't have access to Meta's multimodal models ๐ฟ
๐ค So I thought: why not challenge the small 3B text model with Agentic RAG?
๐ฏ The plan:
- Build a system that tries to answer questions using a knowledge base.
- If the documents don't contain the answer, use Web search for additional context.
Check out my experimental notebook here: ๐ https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/llama32_agentic_rag.ipynb
My stack:
๐๏ธ haystack (https://haystack.deepset.ai/): open-source LLM orchestration framework
๐ฆ meta-llama/Llama-3.2-3B-Instruct
๐ฆ๐ free DuckDuckGo API, integrated with Haystack
โจ ๐๐ฉ๐ฆ ๐ณ๐ฆ๐ด๐ถ๐ญ๐ต๐ด? ๐๐ฏ๐ค๐ฐ๐ถ๐ณ๐ข๐จ๐ช๐ฏ๐จ - ๐ข ๐ง๐ฆ๐ธ ๐ฎ๐ฐ๐ฏ๐ต๐ฉ๐ด ๐ข๐จ๐ฐ, ๐ต๐ฉ๐ช๐ด ๐ญ๐ฆ๐ท๐ฆ๐ญ ๐ฐ๐ง ๐ฑ๐ฆ๐ณ๐ง๐ฐ๐ณ๐ฎ๐ข๐ฏ๐ค๐ฆ ๐ง๐ณ๐ฐ๐ฎ ๐ข ๐ด๐ฎ๐ข๐ญ๐ญ ๐ฎ๐ฐ๐ฅ๐ฆ๐ญ ๐ธ๐ฐ๐ถ๐ญ๐ฅ'๐ท๐ฆ ๐ฃ๐ฆ๐ฆ๐ฏ ๐ถ๐ฏ๐ต๐ฉ๐ช๐ฏ๐ฌ๐ข๐ฃ๐ญ๐ฆ!
This probably reflects the impressive IFEval score of the model (comparable to Llama 3.1 8B).
I was excited to explore Llama 3.2, but as a simple ๐ช๐บ EU guy, I don't have access to Meta's multimodal models ๐ฟ
๐ค So I thought: why not challenge the small 3B text model with Agentic RAG?
๐ฏ The plan:
- Build a system that tries to answer questions using a knowledge base.
- If the documents don't contain the answer, use Web search for additional context.
Check out my experimental notebook here: ๐ https://colab.research.google.com/github/deepset-ai/haystack-cookbook/blob/main/notebooks/llama32_agentic_rag.ipynb
My stack:
๐๏ธ haystack (https://haystack.deepset.ai/): open-source LLM orchestration framework
๐ฆ meta-llama/Llama-3.2-3B-Instruct
๐ฆ๐ free DuckDuckGo API, integrated with Haystack
โจ ๐๐ฉ๐ฆ ๐ณ๐ฆ๐ด๐ถ๐ญ๐ต๐ด? ๐๐ฏ๐ค๐ฐ๐ถ๐ณ๐ข๐จ๐ช๐ฏ๐จ - ๐ข ๐ง๐ฆ๐ธ ๐ฎ๐ฐ๐ฏ๐ต๐ฉ๐ด ๐ข๐จ๐ฐ, ๐ต๐ฉ๐ช๐ด ๐ญ๐ฆ๐ท๐ฆ๐ญ ๐ฐ๐ง ๐ฑ๐ฆ๐ณ๐ง๐ฐ๐ณ๐ฎ๐ข๐ฏ๐ค๐ฆ ๐ง๐ณ๐ฐ๐ฎ ๐ข ๐ด๐ฎ๐ข๐ญ๐ญ ๐ฎ๐ฐ๐ฅ๐ฆ๐ญ ๐ธ๐ฐ๐ถ๐ญ๐ฅ'๐ท๐ฆ ๐ฃ๐ฆ๐ฆ๐ฏ ๐ถ๐ฏ๐ต๐ฉ๐ช๐ฏ๐ฌ๐ข๐ฃ๐ญ๐ฆ!
This probably reflects the impressive IFEval score of the model (comparable to Llama 3.1 8B).
posted
an
update
3 months ago
Post
1082
๐๐ฒ ๐๐ข๐ซ๐ฌ๐ญ ๐๐จ๐ฆ๐ฆ๐ฎ๐ง๐ข๐ญ๐ฒ ๐๐ซ๐ญ๐ข๐๐ฅ๐! ๐๐๐ฅ๐๐๐ญ๐ข๐ฏ๐ ๐๐ข๐ง๐-๐ญ๐ฎ๐ง๐ข๐ง๐ ๐ฐ๐ข๐ญ๐ก ๐๐ฉ๐๐๐ญ๐ซ๐ฎ๐ฆ ๐ฏ
Full walkthrough on how to get started with Spectrum and TRL for efficient fine-tuning.
๐ ๐ฃ https://huggingface.co/blog/anakin87/spectrum
---
Looking to fine-tune Language Models efficiently and save on computational resources?
One popular method is QLoRa, which quantizes the original model and trains low-rank adapters on top.
It's quite effective and uses less GPU than full fine-tuning.
However, QLoRa applies Low-Rank Adaptation uniformly across the entire model.
What if we could identify the most informative layers and only fine-tune those? ๐ค
This is exactly what Spectrum does! ๐
๐ฌ Spectrum analyzes the weight matrices for all layers in a Language Model and calculates a Signal to Noise Ratio (SNR) for each one.
(It uses Random Matrix Theory and Marchenko-Pastur distribution to distinguish signal from noise.)
๐ฏ Based on a chosen percentage (say, 25%), Spectrum selects the most informative layers of each type (mlp.down_proj, self_attn.o_proj, etc.).
You can then โ๏ธ freeze the rest of the model and focus your ๐๏ธโโ๏ธ training on the chosen layers.
๐ Results/Evaluation
- Spectrum is competitive with full fine-tuning and beats QLoRA on benchmarks.
- While QLoRA is more memory-efficient on a single GPU, Spectrum shines in distributed training setups.
- Great models trained with Spectrum: Dolphin models, Llama 3.1 Storm, numerous models by VAGO Solutions...
---
For a practical guide, check out the article above.
Full walkthrough on how to get started with Spectrum and TRL for efficient fine-tuning.
๐ ๐ฃ https://huggingface.co/blog/anakin87/spectrum
---
Looking to fine-tune Language Models efficiently and save on computational resources?
One popular method is QLoRa, which quantizes the original model and trains low-rank adapters on top.
It's quite effective and uses less GPU than full fine-tuning.
However, QLoRa applies Low-Rank Adaptation uniformly across the entire model.
What if we could identify the most informative layers and only fine-tune those? ๐ค
This is exactly what Spectrum does! ๐
๐ฌ Spectrum analyzes the weight matrices for all layers in a Language Model and calculates a Signal to Noise Ratio (SNR) for each one.
(It uses Random Matrix Theory and Marchenko-Pastur distribution to distinguish signal from noise.)
๐ฏ Based on a chosen percentage (say, 25%), Spectrum selects the most informative layers of each type (mlp.down_proj, self_attn.o_proj, etc.).
You can then โ๏ธ freeze the rest of the model and focus your ๐๏ธโโ๏ธ training on the chosen layers.
๐ Results/Evaluation
- Spectrum is competitive with full fine-tuning and beats QLoRA on benchmarks.
- While QLoRA is more memory-efficient on a single GPU, Spectrum shines in distributed training setups.
- Great models trained with Spectrum: Dolphin models, Llama 3.1 Storm, numerous models by VAGO Solutions...
---
For a practical guide, check out the article above.
reacted to
grimjim's
post with ๐
3 months ago
Post
3227
I found this paper to be thought-provoking: "Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling" by Bansal, Hosseini, Agarwal, Tran, and Kazemi.
https://arxiv.org/abs/2408.16737
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.
https://arxiv.org/abs/2408.16737
The direct implication is that smaller models could be used to create cost-effective synthetic datasets. And on that note, in the Gemma terms of use, Google explicitly claims no rights on outputs generated from those models, which means one is free to synthgen from the Gemma line. Meta's Llama 3 licence forbids synthetic generation of outputs if used to improve other models. Relevant Mistral, Qwen, and Yi models under the Apache 2.0 license are unrestricted for this purpose.
- 2 replies
replied to
their
post
3 months ago
This comment has been hidden
posted
an
update
3 months ago
Post
1623
๐ฌ ๐ฎ๐น Phi 3.5 mini ITA: a Small Language Model for Italian
Lately, I've spent some time fine-tuning language models.
Now I am happy to release Phi 3.5 mini ITA: a fine-tuned version of Phi-3.5-mini-instruct to improve performance on the Italian language
๐น Small (3.82 B parameters) but capable model
๐น 128k context length
Chat with it on ๐ค Spaces: anakin87/Phi-3.5-mini-ITA
Model card: anakin87/Phi-3.5-mini-ITA
๐๏ธ Data
Supervised fine-tuning using a good mix of English and Italian data:
- mlabonne/FineTome-100k by @mlabonne
- efederici/capybara-claude-15k-ita by @efederici
๐ Thanks to the authors for the datasets.
๐ฏ Targeted training with Spectrum
I used Spectrum, a relatively new technique for parameter-efficient learning.
The idea is to train only the layers of the model with high Signal-to-Noise Ratio (SNR) and โ๏ธ freeze the rest.
I trained the top 30% of model layers.
๐ Spectrum paper: https://arxiv.org/abs/2406.06623
๐ Vibe check and performance on Italian benchmarks seem encouraging
Lately, I've spent some time fine-tuning language models.
Now I am happy to release Phi 3.5 mini ITA: a fine-tuned version of Phi-3.5-mini-instruct to improve performance on the Italian language
๐น Small (3.82 B parameters) but capable model
๐น 128k context length
Chat with it on ๐ค Spaces: anakin87/Phi-3.5-mini-ITA
Model card: anakin87/Phi-3.5-mini-ITA
๐๏ธ Data
Supervised fine-tuning using a good mix of English and Italian data:
- mlabonne/FineTome-100k by @mlabonne
- efederici/capybara-claude-15k-ita by @efederici
๐ Thanks to the authors for the datasets.
๐ฏ Targeted training with Spectrum
I used Spectrum, a relatively new technique for parameter-efficient learning.
The idea is to train only the layers of the model with high Signal-to-Noise Ratio (SNR) and โ๏ธ freeze the rest.
I trained the top 30% of model layers.
๐ Spectrum paper: https://arxiv.org/abs/2406.06623
๐ Vibe check and performance on Italian benchmarks seem encouraging
-
- 2 replies
reacted to
efederici's
post with โค๏ธ
3 months ago
Post
1551
Finally, I can post! ๐
I created a Capybara-inspired Italian dataset by translating the initial instruction and running it through a pipeline to generate conversations. I used Claude Sonnet for translation and instruction generation, and Opus for generating the answers.
I hope this dataset proves useful for people working on ๐ฎ๐น language models.
โ Open sourcing the dataset here: efederici/capybara-claude-15k-ita
I created a Capybara-inspired Italian dataset by translating the initial instruction and running it through a pipeline to generate conversations. I used Claude Sonnet for translation and instruction generation, and Opus for generating the answers.
I hope this dataset proves useful for people working on ๐ฎ๐น language models.
โ Open sourcing the dataset here: efederici/capybara-claude-15k-ita
- 1 reply
reacted to
gabrielmbmb's
post with โค๏ธ
4 months ago
Post
2868
distilabel 1.3.0 is out! This release contains many core improvements and new tasks that help us building
argilla/magpie-ultra-v0.1!
Distributed pipeline execution with Ray, new Magpie tasks, reward models, components for dataset diversity based on sentence embeddings, Argilla 2.0 compatibility and many more features!
Check the new release in GitHub: https://github.com/argilla-io/distilabel
Distributed pipeline execution with Ray, new Magpie tasks, reward models, components for dataset diversity based on sentence embeddings, Argilla 2.0 compatibility and many more features!
Check the new release in GitHub: https://github.com/argilla-io/distilabel
reacted to
Ameeeee's
post with ๐ฅ
4 months ago
Post
3534
โค๏ธโ๐ฅย Just released version 2.0 of Argilla!
This small revolution includes:
๐ย You can now integrate with the Hugging Face Hub and get started in under five minutes.
๐ชย A single
๐งย Itโs 100 times simpler to configure your dataset now with the new SDK!
๐ย The documentation has been revamped to be cleaner and more user-friendly.
๐ย A new feature automates splitting annotation tasks among a team.
โ๏ธย The layout has been made more flexible to accommodate many use cases.
Check out the release highlights for more details: https://github.com/argilla-io/argilla/releases/tag/v2.0.0
This small revolution includes:
๐ย You can now integrate with the Hugging Face Hub and get started in under five minutes.
๐ชย A single
Dataset class is now designed to handle multiple tasks.๐งย Itโs 100 times simpler to configure your dataset now with the new SDK!
๐ย The documentation has been revamped to be cleaner and more user-friendly.
๐ย A new feature automates splitting annotation tasks among a team.
โ๏ธย The layout has been made more flexible to accommodate many use cases.
Check out the release highlights for more details: https://github.com/argilla-io/argilla/releases/tag/v2.0.0
- 1 reply
reacted to
merve's
post with ๐๐โค๏ธ
4 months ago
Post
3656
At Hugging Face we have an open-source Cookbook with many applied AI recipes ๐
Here are some of the latest recipes contributed โฅฅ
- "Information Extraction with Haystack and NuExtract": Use Haystack and transformers to build structured data extraction pipelines using LLMs by @anakin87 https://huggingface.co/learn/cookbook/en/information_extraction_haystack_nuextract
- "Build RAG with Hugging Face and Milvus": Learn how to use Milvus with sentence transformers to build RAG pipelines https://huggingface.co/learn/cookbook/rag_with_hf_and_milvus
- "Code Search with Vector Embeddings and Qdrant": Search a codebase by building a retrieval pipeline using Qdrant and sentence transformers https://huggingface.co/learn/cookbook/code_search
- Data analyst agent: get your dataโs insights in the blink of an eye โจ: great recipe by our own @m-ric showing how to build an agent that can do data analysis! ๐ฑ https://huggingface.co/learn/cookbook/agent_data_analyst
Here are some of the latest recipes contributed โฅฅ
- "Information Extraction with Haystack and NuExtract": Use Haystack and transformers to build structured data extraction pipelines using LLMs by @anakin87 https://huggingface.co/learn/cookbook/en/information_extraction_haystack_nuextract
- "Build RAG with Hugging Face and Milvus": Learn how to use Milvus with sentence transformers to build RAG pipelines https://huggingface.co/learn/cookbook/rag_with_hf_and_milvus
- "Code Search with Vector Embeddings and Qdrant": Search a codebase by building a retrieval pipeline using Qdrant and sentence transformers https://huggingface.co/learn/cookbook/code_search
- Data analyst agent: get your dataโs insights in the blink of an eye โจ: great recipe by our own @m-ric showing how to build an agent that can do data analysis! ๐ฑ https://huggingface.co/learn/cookbook/agent_data_analyst
reacted to
giux78's
post with โค๏ธ
4 months ago
Post
1639
We https://mii-llm.ai just released a new LLM Italian benchmark and a set of evaluation: MMLU-PRO-ITA
Thanks to @efederici who released efederici/MMLU-Pro-ita a machine translated version of MMLU-PRO and thanks to a community shared computational effort we published in the "Eval Aggiuntive" tab of https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard the results on Italian open source LLMs.
If you want to deepen read the blog article on hf https://huggingface.co/blog/giux78/mmlu-pro-ita
Thanks to @efederici who released efederici/MMLU-Pro-ita a machine translated version of MMLU-PRO and thanks to a community shared computational effort we published in the "Eval Aggiuntive" tab of https://huggingface.co/spaces/FinancialSupport/open_ita_llm_leaderboard the results on Italian open source LLMs.
If you want to deepen read the blog article on hf https://huggingface.co/blog/giux78/mmlu-pro-ita
Nice!
Have a look at my rap model (built with the same approach as MopeyMule): https://huggingface.co/anakin87/yo-Llama-3-8B-Instruct
reacted to
grimjim's
post with โค๏ธ
5 months ago
Post
2262
Below we experiment with negative merger weighting (-1.0!) using task arithmetic. Merge formula on the model card and in the repo itself.
This model is steered to behave opposite to what MopeyMule demonstrated.
Based on the implications of the merge technique, we also propose Orthogonalized Vector Adaptation (OVA). We also extract a LoRA of the counter-refusal abliteration steering vector.
The resulting merger is not a perfect model, but it's a behaviorally interesting model. The model name was inspired by a Philip K. Dick story.
grimjim/Llama-3-Perky-Pat-Instruct-8B
Refusal vector weights ready for use:
grimjim/Llama-3-Instruct-abliteration-OVA-8B
grimjim/Llama-3-Instruct-abliteration-LoRA-8B
This model is steered to behave opposite to what MopeyMule demonstrated.
Based on the implications of the merge technique, we also propose Orthogonalized Vector Adaptation (OVA). We also extract a LoRA of the counter-refusal abliteration steering vector.
The resulting merger is not a perfect model, but it's a behaviorally interesting model. The model name was inspired by a Philip K. Dick story.
grimjim/Llama-3-Perky-Pat-Instruct-8B
Refusal vector weights ready for use:
grimjim/Llama-3-Instruct-abliteration-OVA-8B
grimjim/Llama-3-Instruct-abliteration-LoRA-8B
-
- 3 replies
reacted to
mrm8488's
post with โค๏ธ
5 months ago
Post
4349
๐จExciting news for the Multilingual Synthetic Data Community!๐จ
Iโve taken inspiration from the MAGPIE paper on Llama-3-8B-instruct and extended its capabilities. Hereโs whatโs new!
๐ The MAGPIE paper showcased that if you use the instruction-tuned version (
๐ค While reading a script by Sebastian Raschka, PhD, I wondered: Could these advancements be replicated in other languages? Specifically, could they benefit non-English datasets?
๐ And the answer is YES! At least for Spanish. I've successfully adapted the techniques for Spanish, proving the model's flexibility and multilingual capabilities.
๐ฉโ๐ป To make this accessible, I created a basic script (heavily inspired by the Sebastian Raschka one) that allows you to generate similar datasets using
[Script](https://gist.github.com/mrm8488/4650a5e3cc45523798a527a3446eb312)
๐ Explore the datasets ๐ generated using our new script!
- [Llama-3-8B](https://huggingface.co/datasets/mrm8488/dataset_llama3_5000_samples_es_4231_filtered)
- [Phi-3-medium](https://huggingface.co/datasets/mrm8488/dataset_phi3-medium_5000_samples_es_3906_filtered)
- [Phi-3-mini](https://huggingface.co/datasets/mrm8488/dataset_phi3_5000_samples_es_3282_filtered)
Note: These datasets have basic filtering. Apply additional quality filters before using them to fine-tune large language models.
Inspiration and base script:
https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/05_dataset-generation/llama3-ollama.ipynb
https://www.linkedin.com/feed/update/urn:li:activity:7210982019751661568/
Iโve taken inspiration from the MAGPIE paper on Llama-3-8B-instruct and extended its capabilities. Hereโs whatโs new!
๐ The MAGPIE paper showcased that if you use the instruction-tuned version (
Llama-3-8B-instruct) to generate synthetic instructions and then fine-tune the base version (Llama-3-8B) on this dataset, you can improve even the it-tuned version๐ค While reading a script by Sebastian Raschka, PhD, I wondered: Could these advancements be replicated in other languages? Specifically, could they benefit non-English datasets?
๐ And the answer is YES! At least for Spanish. I've successfully adapted the techniques for Spanish, proving the model's flexibility and multilingual capabilities.
๐ฉโ๐ป To make this accessible, I created a basic script (heavily inspired by the Sebastian Raschka one) that allows you to generate similar datasets using
ollama models (initially phi and llama3) automatically and upload it to the Hugging Face Hub
๐ Explore the datasets ๐ generated using our new script!
- [Llama-3-8B](https://huggingface.co/datasets/mrm8488/dataset_llama3_5000_samples_es_4231_filtered)
- [Phi-3-medium](https://huggingface.co/datasets/mrm8488/dataset_phi3-medium_5000_samples_es_3906_filtered)
- [Phi-3-mini](https://huggingface.co/datasets/mrm8488/dataset_phi3_5000_samples_es_3282_filtered)
Note: These datasets have basic filtering. Apply additional quality filters before using them to fine-tune large language models.
Inspiration and base script:
https://github.com/rasbt/LLMs-from-scratch/blob/main/ch07/05_dataset-generation/llama3-ollama.ipynb
https://www.linkedin.com/feed/update/urn:li:activity:7210982019751661568/
- 7 replies
posted
an
update
5 months ago
Post
1034
How to alter the behavior of a Language Model without fine-tuning or prompting? Say hello to ๐ค yo-Llama ๐ฆ!
Model anakin87/yo-Llama-3-8B-Instruct
This experiment steers Llama-3-8B-Instruct to respond in a rap style.
How? Amplifying the rap direction in the activation space. ๐
๐๐ก๐๐ญ ๐ฌ๐ฉ๐๐ซ๐ค๐๐ ๐ญ๐ก๐ข๐ฌ ๐ข๐๐๐?
Lately, I got interested in mechanistic interpretability of LLMs.
๐ก A recent paper, "Refusal in Language Models Is Mediated by a Single Direction," showed how to find the refusal direction in the activation space of Chat Language Models and either erase or amplify it.
A clever jailbreak method for open weights models.
Then, @failspy took it a step further by modifying the models to amplify different traits, such as making a model seem grumpy or irritable.
๐๐จ๐ฐ ๐๐ข๐ ๐ ๐๐ซ๐๐๐ญ๐ ๐ฒ๐จ-๐๐ฅ๐๐ฆ๐?
(๐ notebook in the HF repository, heavily inspired by Failspy's work)
1๏ธโฃ Load the Llama-3-8B-Instruct model.
2๏ธโฃ Load 1024 examples from Alpaca (instruction dataset).
3๏ธโฃ Prepare a system prompt to make the original model act like a rapper.
4๏ธโฃ Run inference on the examples, with and without the system prompt, and cache the activations.
5๏ธโฃ Compute the rap feature directions (one for each layer) from the activations.
6๏ธโฃ Apply the feature directions one by one, checking the results on some examples.
7๏ธโฃ Pick the best-performing feature direction.
8๏ธโฃ Apply this feature direction and voilร !
yo-Llama-3-8B-Instruct is born! ๐ฅณ๐ถ
This was a fun experiment.
๐ Resources
Refusal in Language Models Is Mediated by a Single Direction - https://arxiv.org/abs/2406.11717
Uncensor any LLM with abliteration: great practical blog post by @mlabonne https://huggingface.co/blog/mlabonne/abliteration
Practical materials by @failspy
- abliterator library https://github.com/FailSpy/abliterator
- Llama-MopeyMule-3-8B-Instruct model (+ notebook) failspy/Llama-3-8B-Instruct-MopeyMule
Model anakin87/yo-Llama-3-8B-Instruct
This experiment steers Llama-3-8B-Instruct to respond in a rap style.
How? Amplifying the rap direction in the activation space. ๐
๐๐ก๐๐ญ ๐ฌ๐ฉ๐๐ซ๐ค๐๐ ๐ญ๐ก๐ข๐ฌ ๐ข๐๐๐?
Lately, I got interested in mechanistic interpretability of LLMs.
๐ก A recent paper, "Refusal in Language Models Is Mediated by a Single Direction," showed how to find the refusal direction in the activation space of Chat Language Models and either erase or amplify it.
A clever jailbreak method for open weights models.
Then, @failspy took it a step further by modifying the models to amplify different traits, such as making a model seem grumpy or irritable.
๐๐จ๐ฐ ๐๐ข๐ ๐ ๐๐ซ๐๐๐ญ๐ ๐ฒ๐จ-๐๐ฅ๐๐ฆ๐?
(๐ notebook in the HF repository, heavily inspired by Failspy's work)
1๏ธโฃ Load the Llama-3-8B-Instruct model.
2๏ธโฃ Load 1024 examples from Alpaca (instruction dataset).
3๏ธโฃ Prepare a system prompt to make the original model act like a rapper.
4๏ธโฃ Run inference on the examples, with and without the system prompt, and cache the activations.
5๏ธโฃ Compute the rap feature directions (one for each layer) from the activations.
6๏ธโฃ Apply the feature directions one by one, checking the results on some examples.
7๏ธโฃ Pick the best-performing feature direction.
8๏ธโฃ Apply this feature direction and voilร !
yo-Llama-3-8B-Instruct is born! ๐ฅณ๐ถ
This was a fun experiment.
๐ Resources
Refusal in Language Models Is Mediated by a Single Direction - https://arxiv.org/abs/2406.11717
Uncensor any LLM with abliteration: great practical blog post by @mlabonne https://huggingface.co/blog/mlabonne/abliteration
Practical materials by @failspy
- abliterator library https://github.com/FailSpy/abliterator
- Llama-MopeyMule-3-8B-Instruct model (+ notebook) failspy/Llama-3-8B-Instruct-MopeyMule
reacted to
fdaudens's
post with โค๏ธ
5 months ago
Post
1871
Look at that ๐
Actual benchmarks have become too easy for recent models, much like grading high school students on middle school problems makes little sense. So the team worked on a new version of the Open LLM Leaderboard with new benchmarks.
Stellar work from @clefourrier @SaylorTwift and the team!
๐ Read the blog post: open-llm-leaderboard/blog
๐ Explore the leaderboard: open-llm-leaderboard/open_llm_leaderboard
Actual benchmarks have become too easy for recent models, much like grading high school students on middle school problems makes little sense. So the team worked on a new version of the Open LLM Leaderboard with new benchmarks.
Stellar work from @clefourrier @SaylorTwift and the team!
๐ Read the blog post: open-llm-leaderboard/blog
๐ Explore the leaderboard: open-llm-leaderboard/open_llm_leaderboard
- 1 reply