Improving Hugging Face Training Efficiency Through Packing with Flash Attention
•
22
Mayank Mishra
mayank-mishra
AI & ML interests
Large Language Models, Distributed Training and Inference
Recent Activity
upvoted
a
paper
20 days ago
upvoted
a
collection
21 days ago
SmolLM2
New activity
29 days ago
ibm-granite/granite-3.0-2b-instruct
Articles
Organizations
Posts
4
Post
1736
New preprint out with colleagues from MIT and IBM Research
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Reducing Transformer Key-Value Cache Size with Cross-Layer Attention (2405.12981)
We introduce a simple mechanism of sharing keys and values across layers, reducing the memory needed for KV cache during inference!!
Post
2500
Thrilled to unveil DS-MoE: a dense training and sparse inference scheme for enhanced computational and memory efficiency in your MoE models! 🚀🚀🚀
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Discover more in our blog: https://huggingface.co/blog/bpan/ds-moe and dive into the details with our paper: Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models (2404.05567)
Collections
1
models
8
mayank-mishra/granite-3b-code-glaive-20k
Text Generation
•
Updated
•
8
mayank-mishra/granite-20b-code-instruct-Q4_K_M-GGUF
Text Generation
•
Updated
•
10
mayank-mishra/starcoder-GPTQ-8bit-128g
Updated
•
11
mayank-mishra/starcoder-GPTQ-4bit-128g
Updated
•
16
mayank-mishra/starcoderbase-GPTQ-4bit-128g
Updated
•
21
mayank-mishra/starcoderbase-GPTQ-8bit-128g
Updated
•
3
mayank-mishra/santacoder-GPTQ-4bit-128g
Updated
•
2
mayank-mishra/santacoder-GPTQ-8bit-128g
Updated
•
1