4 bit and 8 bit bnb quants only generate empty strings or one token repeated endlessly

#32

by nicorinn-google - opened Jul 24

Jul 24

I'm using the same exact notebook for the 27b-it and 9b-it versions, so the issue is definitely related to this model. Any ideas of what the cause may be?

mdouglas

Aug 1

Hi @nicorinn-google , please use torch_dtype=torch.bfloat16 when loading with from_pretrained(). There's a PR to update the model card examples here: #33.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment