Congrats!
Kaito Sugimoto
kaisugi
AI & ML interests
Japanese LLMs
Recent Activity
liked
a Space
1 day ago
llm-jp/open-japanese-llm-leaderboard
replied to
AkimfromParis's
post
1 day ago
reacted to
AkimfromParis's
post
with 👍
1 day ago
Organizations
kaisugi's activity
replied to
AkimfromParis's
post
1 day ago
reacted to
AkimfromParis's
post with 👍
1 day ago
Post
917
🇯🇵 The Open Japanese LLM Leaderboard created by LLM-jp 🌸 in partnership with HuggingFace 🤗 was released today!
Blog: https://huggingface.co/blog/leaderboard-japanese
Space: llm-jp/open-japanese-llm-leaderboard
🌍 The leaderboard is available in both Japanese and English
📚 Based on the evaluation tool, llm-jp-eval with more than 20 datasets for Japanese LLMs
📊 The leaderboard showcases all the metrics for NLP experts, plus averages for NLP beginners
💻 For the comfort of users, we chose a horizontal UI, and implemented it in a light and dark theme on Gradio
🔬 The radar chart provides a very interesting visualization of metrics!
🌱 We are using the Japanese research platform, MDX, so please be patient!
⚡ LLM bigger than +70B will be evaluated soon…
How do you say “GPUs Go Brrr” in Japanese - > GPUがブンブン～! (To pronounce "GPU ga bunbun!") 🔥
Blog: https://huggingface.co/blog/leaderboard-japanese
Space: llm-jp/open-japanese-llm-leaderboard
🌍 The leaderboard is available in both Japanese and English
📚 Based on the evaluation tool, llm-jp-eval with more than 20 datasets for Japanese LLMs
📊 The leaderboard showcases all the metrics for NLP experts, plus averages for NLP beginners
💻 For the comfort of users, we chose a horizontal UI, and implemented it in a light and dark theme on Gradio
🔬 The radar chart provides a very interesting visualization of metrics!
🌱 We are using the Japanese research platform, MDX, so please be patient!
⚡ LLM bigger than +70B will be evaluated soon…
How do you say “GPUs Go Brrr” in Japanese - > GPUがブンブン～! (To pronounce "GPU ga bunbun!") 🔥
-
- 4 replies
posted
an
update
5 months ago
Post
735
🚀 Llama-3-ELYZA-JP-8B
ELYZA, Inc. has developed two large language models (LLMs) for Japanese called "Llama-3-ELYZA-JP-70B" with 70 billion parameters and "Llama-3-ELYZA-JP-8B" with 8 billion parameters, based on Meta's "Llama 3" series. These models have been fine-tuned through additional pre-training and post-training to improve Japanese language capabilities significantly.
Key Points:
Performance:
- Llama-3-ELYZA-JP-70B surpasses global models such as GPT-4, Claude 3 Sonnet, and Gemini 1.5 Flash.
- Llama-3-ELYZA-JP-8B matches models like GPT-3.5 Turbo and Claude 3 Haiku despite having fewer parameters.
Availability:
- The 8B model is available on Hugging Face Hub and can be used for both research and commercial purposes under the Llama 3 Community License.
Methodology:
- ELYZA enhanced the Japanese performance of the Llama 3 models through additional training with high-quality Japanese corpora and Instruction Tuning with proprietary datasets.
Benchmarks:
- Evaluations using ELYZA Tasks 100 and Japanese MT-Bench showed significant improvements in Japanese language generation.
Inference Speed:
- To address inference speed issues due to model size, ELYZA implemented Speculative Decoding, which achieved up to 1.6 times faster inference for the 70B model.
Overall, ELYZA's models demonstrate state-of-the-art performance in Japanese language tasks and are optimized for both efficiency and effectiveness.
Model URL:
- elyza/Llama-3-ELYZA-JP-8B
- elyza/Llama-3-ELYZA-JP-8B-AWQ
- elyza/Llama-3-ELYZA-JP-8B-GGUF
Blog post (in Japanese):
https://note.com/elyza/n/n360b6084fdbd
ELYZA, Inc. has developed two large language models (LLMs) for Japanese called "Llama-3-ELYZA-JP-70B" with 70 billion parameters and "Llama-3-ELYZA-JP-8B" with 8 billion parameters, based on Meta's "Llama 3" series. These models have been fine-tuned through additional pre-training and post-training to improve Japanese language capabilities significantly.
Key Points:
Performance:
- Llama-3-ELYZA-JP-70B surpasses global models such as GPT-4, Claude 3 Sonnet, and Gemini 1.5 Flash.
- Llama-3-ELYZA-JP-8B matches models like GPT-3.5 Turbo and Claude 3 Haiku despite having fewer parameters.
Availability:
- The 8B model is available on Hugging Face Hub and can be used for both research and commercial purposes under the Llama 3 Community License.
Methodology:
- ELYZA enhanced the Japanese performance of the Llama 3 models through additional training with high-quality Japanese corpora and Instruction Tuning with proprietary datasets.
Benchmarks:
- Evaluations using ELYZA Tasks 100 and Japanese MT-Bench showed significant improvements in Japanese language generation.
Inference Speed:
- To address inference speed issues due to model size, ELYZA implemented Speculative Decoding, which achieved up to 1.6 times faster inference for the 70B model.
Overall, ELYZA's models demonstrate state-of-the-art performance in Japanese language tasks and are optimized for both efficiency and effectiveness.
Model URL:
- elyza/Llama-3-ELYZA-JP-8B
- elyza/Llama-3-ELYZA-JP-8B-AWQ
- elyza/Llama-3-ELYZA-JP-8B-GGUF
Blog post (in Japanese):
https://note.com/elyza/n/n360b6084fdbd
posted
an
update
5 months ago
Post
616
🚀 KARAKURI LM 8x7B Instruct v0.1
KARAKURI Inc. has publicly released "KARAKURI LM 8x7B Instruct v0.1", the first domestic Large Language Model (LLM) in Japan to support Function calling and Retrieval-Augmented Generation (RAG). This AI agent can handle tasks across various applications autonomously, significantly reducing implementation costs compared to traditional models.
Model Features:
- Capable of autonomously choosing optimal documents and databases for various tasks.
- Applied extensively in customer support for automating responses and processes, analyzing Voice of Customer (VoC), and predicting optimal outreach timings.
Model URL:
karakuri-ai/karakuri-lm-8x7b-instruct-v0.1
Detailed press release (in Japanese):
https://karakuri.ai/seminar/news/karakuri-lm-8x7b-instruct-v0-1/
KARAKURI Inc. has publicly released "KARAKURI LM 8x7B Instruct v0.1", the first domestic Large Language Model (LLM) in Japan to support Function calling and Retrieval-Augmented Generation (RAG). This AI agent can handle tasks across various applications autonomously, significantly reducing implementation costs compared to traditional models.
Model Features:
- Capable of autonomously choosing optimal documents and databases for various tasks.
- Applied extensively in customer support for automating responses and processes, analyzing Voice of Customer (VoC), and predicting optimal outreach timings.
Model URL:
karakuri-ai/karakuri-lm-8x7b-instruct-v0.1
Detailed press release (in Japanese):
https://karakuri.ai/seminar/news/karakuri-lm-8x7b-instruct-v0-1/
posted
an
update
5 months ago
Post
2267
🚀 Sarashina1-65B
SB Intuitions has announced the release of Japanese Large Language Models (LLMs) with 7 billion, 13 billion, and 65 billion parameters to aid academic and industrial research and development. The company plans to develop a 390 billion parameter model by the end of 2024. The models, named Sarashina1 and Sarashina2, show significant performance improvements, especially Sarashina2 which is an enhanced version of Sarashina1.
Performance evaluations using five Japanese language datasets reveal that Sarashina2 outperforms other models, including continued pre-trained models. The name "Sarashina" originates from a historical diary linked to the headquarters' location in Tokyo's Takeshiba area, symbolizing the company's ambition to create globally utilized models from Japan.
Model URL:
- sbintuitions/sarashina1-65b
- sbintuitions/sarashina2-13b
Detailed press release (in Japanese):
https://www.sbintuitions.co.jp/news/press/20240614_01/
SB Intuitions has announced the release of Japanese Large Language Models (LLMs) with 7 billion, 13 billion, and 65 billion parameters to aid academic and industrial research and development. The company plans to develop a 390 billion parameter model by the end of 2024. The models, named Sarashina1 and Sarashina2, show significant performance improvements, especially Sarashina2 which is an enhanced version of Sarashina1.
Performance evaluations using five Japanese language datasets reveal that Sarashina2 outperforms other models, including continued pre-trained models. The name "Sarashina" originates from a historical diary linked to the headquarters' location in Tokyo's Takeshiba area, symbolizing the company's ambition to create globally utilized models from Japan.
Model URL:
- sbintuitions/sarashina1-65b
- sbintuitions/sarashina2-13b
Detailed press release (in Japanese):
https://www.sbintuitions.co.jp/news/press/20240614_01/
posted
an
update
5 months ago
Post
869
🚀 llava-calm2-siglip
CyberAgent Inc. has announced the public release of "llava-calm2-siglip," a 7.5 billion parameter Vision Language Model (VLM) for Japanese, available for commercial use. This model, trained primarily on a high-quality Japanese dataset, is accessible on Hugging Face Hub under an Apache-2.0 license. The advancement aims to improve Japanese language-specific VLMs, which are fewer compared to English-centric models.
Model URL:
cyberagent/llava-calm2-siglip
Demo URL:
cyberagent/llava-calm2-preview
Detailed press release (in Japanese): https://www.cyberagent.co.jp/news/detail/id=30344
CyberAgent Inc. has announced the public release of "llava-calm2-siglip," a 7.5 billion parameter Vision Language Model (VLM) for Japanese, available for commercial use. This model, trained primarily on a high-quality Japanese dataset, is accessible on Hugging Face Hub under an Apache-2.0 license. The advancement aims to improve Japanese language-specific VLMs, which are fewer compared to English-centric models.
Model URL:
cyberagent/llava-calm2-siglip
Demo URL:
cyberagent/llava-calm2-preview
Detailed press release (in Japanese): https://www.cyberagent.co.jp/news/detail/id=30344
reacted to
leonardlin's
post with 👍
6 months ago
Post
1942
With slurm figured out and ablations humming along, I though I'd update and post my understanding of the legal status of training data in Japan. It is in general, much clearer in the US: https://huggingface.co/blog/leonardlin/ai-training-data-in-japan
replied to
their
post
6 months ago
That's a good point.
I'm not an employee of this company or working in the financial sector, but I do know that people involved have actively discussed in which case they should make use of LLMs. I guess LLMs won't replace humans' decision-making processes, but rather augment them.
posted
an
update
6 months ago
Post
1579
🚀 Stockmark-100b
Stockmark Inc. has developed and released one of Japan's largest commercial-scale Language Models (LLM) with 100 billion parameters, named "Stockmark-LLM-100b". This model significantly reduces hallucinations and provides accurate responses to complex business-related queries. Developed from scratch with a focus on Japanese business data, the model aims to be reliable for high-stakes business environments. It's open-source and available for commercial use.
Key highlights:
- The model reduces hallucinations—incorrect confident responses that AI models sometimes generate.
- Stockmark-LLM-100b can answer basic business questions and specialized queries in industries like manufacturing.
- The model's performance surpasses GPT-4-turbo in accuracy for business-specific queries.
- Evaluation benchmarks (VicunaQA) show high performance.
- Fast inference speed, generating 100-character Japanese text in 1.86 seconds.
stockmark/stockmark-100b
stockmark/stockmark-100b-instruct-v0.1
Detailed press release (in Japanese): https://stockmark.co.jp/news/20240516
Stockmark Inc. has developed and released one of Japan's largest commercial-scale Language Models (LLM) with 100 billion parameters, named "Stockmark-LLM-100b". This model significantly reduces hallucinations and provides accurate responses to complex business-related queries. Developed from scratch with a focus on Japanese business data, the model aims to be reliable for high-stakes business environments. It's open-source and available for commercial use.
Key highlights:
- The model reduces hallucinations—incorrect confident responses that AI models sometimes generate.
- Stockmark-LLM-100b can answer basic business questions and specialized queries in industries like manufacturing.
- The model's performance surpasses GPT-4-turbo in accuracy for business-specific queries.
- Evaluation benchmarks (VicunaQA) show high performance.
- Fast inference speed, generating 100-character Japanese text in 1.86 seconds.
stockmark/stockmark-100b
stockmark/stockmark-100b-instruct-v0.1
Detailed press release (in Japanese): https://stockmark.co.jp/news/20240516
-
- 4 replies
reacted to
leonardlin's
post with 👍
6 months ago
Post
1362
llm-jp-eval is currently one of the most widely used benchmarks for Japanese LLMs and is half of WandB's comprehensive Nejumi LLM Leaderboard scoring. I was seeing some weirdness in results I was getting and ended up in a bit of a rabbit hole. Here's my article on evaling llm-jp-eval: https://huggingface.co/blog/leonardlin/llm-jp-eval-eval
I've setup a fork of Lightblue's Shaberi testing framework which uses LLM-as-a-Judge style benchmarks as something probably more representative of real world LLM strength in Japanese. Here's how the new base model ablations are looking:
I've setup a fork of Lightblue's Shaberi testing framework which uses LLM-as-a-Judge style benchmarks as something probably more representative of real world LLM strength in Japanese. Here's how the new base model ablations are looking: