Can we use orthogonalization to make the LLaMa-3 (8B-70B) More Intelligent?
#4
by
Joseph717171
- opened
Love your work! I have a question though: can we use orthogonalization to make LLaMa-3 more intelligent in its generations and prose? The thought just occurred to me after your LLaMa-3-MopeyMule Release, and my curiosity was reignited when I stumbled upon your Reddit post, detailing orthogonalization and ablation and your piqued curiosity to see what other purposes they can be used for. I think it would be cool to see orthogonalization used to make LLaMa-3’s generations more intelligent (context aware and with formatting awareness).
You’re work has some similarities to Vgel’s work with control vectors (Vgel’s blog detailing Control vectors)). Perhaps Vgel’s experiments might be helpful and enlightening for you and your work. 😁