Titus von Koeller

Titus-von-Koeller

AI & ML interests

NN Quantization, Generative AI, LLMs, alignment, algorithms for social justice, ethical humanism, mitigating gender bias, audio compression, AGI

Articles

Organizations

Posts 3

view post
Post
1918
πŸ”₯ Level up your model training w/ GaLore + Transformers for SOTA results on consumer-grade hardware!

⬇️ 82.5% less optimizer state memory footprint without performance degradation by expressing the gradient weight matrix as low rank.

πŸ‘©πŸΏβ€πŸ’» Install via pip install transformers>=4.39.0 galore-torch. #ProudlyGpuPoor

The integration of GaLore into the training of large language models (LLMs) marks a significant advancement in the field of deep learning, particularly in terms of memory efficiency and the democratization of AI research. By allowing for the training of billion-parameter models on consumer-grade hardware, reducing memory footprint in optimizer states, and leveraging advanced projection matrix techniques, GaLore opens new horizons for researchers and practitioners with limited access to high-end computational resources.

πŸ”¬ Find out more about GaLore and investigate lots of juicy technical details: https://huggingface.co/blog/galore

πŸ€— Huge thanks to everyone involved ❀️:

β€’ authors: @jiaweizhao @Kyriection @beidic Zhangyang Wang @animakumar @tydsh
β€’ community contributors: @hiyouga @mdouglas and others!
β€’ @ybelkada for taking such swift action in composing and coordinating necessary PRs to get this live at ⚑ speed!

πŸ—οΈπŸ“ˆ Super rewarding to see how @timdettmers work with optimizers is being built upon to achieve even greater heights!

🚧 Actually, there are ongoing works to integrate GaLore into bitsandbytes and optimize memory efficiency even further πŸ’ͺ. We'll keep you posted!
view post
Post
We just released bitsandbytes==0.43.0 πŸ“¦ , with these significant new additions:

β€£ πŸ›« FSDP+QLoRA support (alpha release)
β—¦ now anyone with 2 powerful gaming GPUs can fine-tune 70B param models at home!
β—¦ in collab with Jeremy Howard + team @ answer.ai
β—¦ answer.ai blogpost: https://www.answer.ai/posts/2024-03-06-fsdp-qlora.html
β—¦ example repo: https://github.com/AnswerDotAI/fsdp_qlora/

β€£ 🌈⊞ Official Windows support
β—¦ now via simple pip install bitsandbytes>=0.43.0

β€£ πŸ“„ Huge docs update:
β—¦ https://huggingface.co/docs/bitsandbytes/main
β—¦ Be sure to check out the optimizers and the API docs
β—¦ ... even more upcoming ...

Under the hood there we have many other improvements, due to extensive maintenance activity, community contributions by super active + knowledgable volunteers ✨ πŸš€ and the official sponsorship by Hugging Face that makes all this possible πŸ€— ❀️ 🌍

We would greatly appreciate any further community contributions, be it to help with refactorings, exterminating flaky tests, writing doc-strings, tutorials, new features. Don't be shy, just contact us and we see where this leads us:
https://github.com/TimDettmers/bitsandbytes/discussions

Have a great weekend everyone!

models

None public yet

datasets

None public yet