Edit model card

BLOOM LM - 8bit

BigScience Large Open-science Open-access Multilingual Language Model - 8bit

Model Card

BigScience Logo

Version 1.0 / 26.May.2022

Related paper: https://arxiv.org/abs/2208.07339

TL;DR

This repository contains 8bit weights of bloom-1b7 model. You can load this model using transformers==4.28.0 and bitsandbytes>0.37.2 out of the box !

# pip install accelerate bitsandbytes
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("ybelkada/bloom-1b7-8bit")

How to push 8bit weights?

First, make sure you are using transformers & bitsandbytes versions stated above. Then load your 8bit model as usual using load_in_8bit=True!

# pip install accelerate bitsandbytes
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("bigscience/bloom-1b7", device_map="auto", load_in_8bit=True)

Then just call push_to_hub method or save_pretrained method if you want to save your 8bit model locally

model.push_to_hub("{your_username}/bloom-1b7-8bit")

That's it!

What is inside the model's state_dict?

Inside the state dict of the model (pytorch_model.bin file) you have

  • the quantized int8 weights
  • the quantization statistics in float16
Downloads last month
497
Safetensors
Model size
1.72B params
Tensor type
F32
FP16
I8
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Space using ybelkada/bloom-1b7-8bit 1