Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
fblgit 
posted an update Feb 10
Post
Introducing UNA-SimpleSmaug-34b:

Based on Smaug-34B-v0.1, capable of slightly outperform his base model and with increased math and reasoning thanks to simple-math dataset.
The model exhibits a great performance across diverse tasks with an excellent and balanced behaviour.
It scores 77.41 AVG on the Leaderboard, landing on #1 Position of 34B models.

Available in the hub already:
fblgit/UNA-SimpleSmaug-34b-v1beta
fblgit/simple-math

In this case, we applied UNA to the Attention Layers of the model while performing SFT with simple-math on a high complexity generated data of mathematics, proving the effect of simple-math on LLM's.

Nice! How did you use UNA w/ Axolotl?

·

UNA is a modification of the modeling_$model.py of transformers. I port it to to the different transformer version and models, keeping it clean and performant, So it works with any of these frameworks like #axolotl

In this post