Aria: An Open Multimodal Native Mixture-of-Experts Model
teowu
Abstract
Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.
Community
Thanks for this great release!
Here's my summary:
Rhymes AI drops Aria: small Multimodal MoE that beats GPT-4o and Gemini-1.5-Flash ⚡️
New player entered the game! Rhymes AI has just been announced, and unveiled Aria – a multimodal powerhouse that's punching above its weight.
Key insights:
🧠 Mixture-of-Experts architecture: 25.3B total params, but only 3.9B active.
🌈 Multimodal: text/image/video → text.
📚 Novel training approach: “multimodal-native” where multimodal training starts directly during pre-training, not just tacked on later
📏 Long 64K token context window
🔓 Apache 2.0 license, with weights, code, and demos all open
⚡️ On the benchmark side, Aria leaves some big names in the dust.
- It beats Pixtral 12B or Llama-3.2-12B on several vision benchmarks like MMMU or MathVista.
- It even overcomes the much bigger GPT-4o on long video tasks and even outshines Gemini 1.5 Flash when it comes to parsing lengthy documents.
But Rhymes AI isn't just showing off benchmarks. They've already got Aria powering a real-world augmented search app called “Beago”. It’s handling even recent events with great accuracy!
And they partnered with AMD to make it much faster than competitors like Perplexity or Gemini search. 🚀
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MM1.5: Methods, Analysis&Insights from Multimodal LLM Fine-tuning (2024)
- NVLM: Open Frontier-Class Multimodal LLMs (2024)
- xGen-MM (BLIP-3): A Family of Open Large Multimodal Models (2024)
- Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Multimodal Models (2024)
- Emu3: Next-Token Prediction is All You Need (2024)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Amazing. Thanks!
Please extract all text into JSON format
This paper was a great read. We wrote a summary blog about this paper and a few more like
- TPI LLM
- Differential Transformer
- ARIA
You can find it here. Please give it a read :)
Models citing this paper 1
Datasets citing this paper 0
No dataset linking this paper