TheBloke

/

stable-vicuna-13B-HF

like 95

Yeah this is an issue with this model. Check out this code by Sam Witteveen - he implements a bit of extra code to split on ### Human : https://colab.research.google.com/drive/1Kvf3qF1TXE-jR-N5G9z1XxVf5z-ljFt2?usp=sharing

import json
import textwrap

human_prompt = 'What is the meaning of life?'

def get_prompt(human_prompt):
    prompt_template=f"### Human: {human_prompt} \n### Assistant:"
    return prompt_template

print(get_prompt('What is the meaning of life?'))

def remove_human_text(text):
    return text.split('### Human:', 1)[0]

def parse_text(data):
    for item in data:
        text = item['generated_text']
        assistant_text_index = text.find('### Assistant:')
        if assistant_text_index != -1:
            assistant_text = text[assistant_text_index+len('### Assistant:'):].strip()
            assistant_text = remove_human_text(assistant_text)
            wrapped_text = textwrap.fill(assistant_text, width=100)
            print(wrapped_text)

data = [{'generated_text': '### Human: What is the capital of England? \n### Assistant: The capital city of England is London.'}]
parse_text(data)

Thank you very much. Just to clarify, it still has the problem that it will take longer to generate right?

Yes, but I think that's just how it is with this model. If that's a deal breaker for you, try WizardLM 7B or Vicuna 1.1 13B.

got it, thank you!

Will there be a 1.1 version of the stable vicuna?

Will there be a 1.1 version of the stable vicuna?

Check out this - it is Wizard dataset using Vicuna 1.1 training method, on 13B. People are saying it's really good:

https://huggingface.co/TheBloke/wizard-vicuna-13B-HF
https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ
https://huggingface.co/TheBloke/wizard-vicuna-13B-GGML

Will try

Will there be a 1.1 version of the stable vicuna?

Check out this - it is Wizard dataset using Vicuna 1.1 training method, on 13B. People are saying it's really good:

https://huggingface.co/TheBloke/wizard-vicuna-13B-HF
https://huggingface.co/TheBloke/wizard-vicuna-13B-GPTQ
https://huggingface.co/TheBloke/wizard-vicuna-13B-GGML

Happy to hear that. Thank you

I finally found a way to fix the issue. I adopted the _SentinelTokenStoppingCriteria class from this repo: https://github.com/oobabooga/text-generation-webui/blob/2cf711f35ec8453d8af818be631cb60447e759e2/modules/callbacks.py#L12. And then pass the stop_word token ids to the _SentinelTokenStoppingCriteria class. You can use "\n###" and/or "\n### Human:" as stop words. But somehow, the tokenizer will automatically add the "_" token on the left if you encode a string that starts with "\n". So, the first token_id need to be removed from the token_id tensor. You will also need to remove the stop words from the final text output. Here is the code snippet for the fix:

stop_words = ["</s>",  "\n###", "\n### Human:"]

stopping_criteria_list = StoppingCriteriaList()

sentinel_token_ids = []
for string in stop_words:
    if string.startswith("\n"):
        sentinel_token_ids.append(
            self.tokenizer.encode(
                string, return_tensors="pt", add_special_tokens=False
            )[:, 1:].to(self.device)
        )
    else:
        sentinel_token_ids.append(
            self.tokenizer.encode(
                string, return_tensors="pt", add_special_tokens=False
            ).to(self.device)
        )

stopping_criteria_list.append(
            _SentinelTokenStoppingCriteria(
                sentinel_token_ids=sentinel_token_ids, starting_idx=len(input_ids[0])
            )
        )

gen_tokens = self.model.generate(
            input_ids,
            stopping_criteria=stopping_criteria_list,
            **_model_kwargs
        )

Good to know that wizard-vicuna-13B-HF is available and free from this issue. I will definitely give it a try!