NickyNicky/gemma-2b-it_oasst2_chatML_Cluster2_aya_multilingual

reference data model:

  datasets:
    - lang: "bg,ca,cs,da,de,en,es,fr,hr,hu,it,nl,pl,pt,ro,ru,sl,sr,sv,uk"
      link: https://huggingface.co/datasets/NickyNicky/oasst2_clusters

  model:
    - google/gemma-2b-it
      Link:
        https://huggingface.co/google/gemma-2b-it

    base fine tune: NickyNicky/gemma-2b-it_oasst2_chatML_Cluster_2_V1

  Epoch: 2

  future experts: 6

  Eval model:
    - link:
        soon

train/loss 0.8529

!python -m pip install --upgrade pip
!pip install "torch>=2.1.1" -U
!pip install torchaudio==2.2.0
!pip install -q datasets trl peft bitsandbytes sentencepiece wandb
!pip install -q accelerate safetensors deepspeed
!pip install -q scipy ninja -U
!pip install -q -U transformers==4.38.0
!pip install flash-attn==2.5.5 --no-build-isolation

Version

import torch
torch.__version__
#OUTPUTS: ('2.2.0+cu121' )

How to use

from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
    HfArgumentParser,
    TrainingArguments,
    pipeline,
    logging,
    GenerationConfig,
    TextIteratorStreamer,
)

from transformers import StoppingCriteria, StoppingCriteriaList

import torch

model_id='NickyNicky/gemma-2b-it_oasst2_chatML_Cluster2_aya_multilingual'

model = AutoModelForCausalLM.from_pretrained(model_id,
                                             device_map="auto",
                                             trust_remote_code=True,
                                             torch_dtype=torch.bfloat16,
                                             attn_implementation="flash_attention_2",
                                             # load_in_4bit=True,
                                             # low_cpu_mem_usage= True,

                                             )

max_length=1355
print("max_length",max_length)


tokenizer = AutoTokenizer.from_pretrained(model_id,
                                          # use_fast = False,
                                          max_length=max_length,)


class ListOfTokensStoppingCriteria(StoppingCriteria):
    """
    Clase para definir un criterio de parada basado en una lista de tokens específicos.
    """
    def __init__(self, tokenizer, stop_tokens):
        self.tokenizer = tokenizer
        # Codifica cada token de parada y guarda sus IDs en una lista
        self.stop_token_ids_list = [tokenizer.encode(stop_token, add_special_tokens=False) for stop_token in stop_tokens]

    def __call__(self, input_ids, scores, **kwargs):
        # Verifica si los últimos tokens generados coinciden con alguno de los conjuntos de tokens de parada
        for stop_token_ids in self.stop_token_ids_list:
            len_stop_tokens = len(stop_token_ids)
            if len(input_ids[0]) >= len_stop_tokens:
                if input_ids[0, -len_stop_tokens:].tolist() == stop_token_ids:
                    return True
        return False

# Uso del criterio de parada personalizado
stop_tokens = ["<end_of_turn>"]  # Lista de tokens de parada

# Inicializa tu criterio de parada con el tokenizer y la lista de tokens de parada
stopping_criteria = ListOfTokensStoppingCriteria(tokenizer, stop_tokens)

# Añade tu criterio de parada a una StoppingCriteriaList
stopping_criteria_list = StoppingCriteriaList([stopping_criteria])


# improves control of responses in different languages.
# trainer language codes: ["es", "en", "fr", "de"]
input_code= "es"
target_code= "en"


#EXAMPLE #1
input_text = f"""<bos><start_of_turn>system
You are a helpful AI assistant.
lista de codigos linguisticos disponibles: ["{input_code}", "{target_code}"].<end_of_turn>
<start_of_turn>user
escribe una historia de 100 palabras<end_of_turn>
<start_of_turn>model
"""

### OUTPUT EXAMPLE
###'''

#<start_of_turn>model
#Once upon a time, in a quiet town nestled between rolling hills and lush forests, there lived a wise old owl named Merlin. Merlin had lived for many years, witnessing the passing of time and the changing seasons.
#One sunny morning, as Merlin was perching on a branch high in the trees, he noticed a strange sight: a strange shadow following him like a ghost. To his surprise, it was a black bird with heavy wings.
#Merlin's eyes widened in surprise and he knew that this bird was a visitor from the beyond. He tried to call upon his magic, but the bird's song couldn't be heard by him. The bird flew away, singing far away, as though Merlin's magic didn't exist.
#That day, the wise owl decided to show the visitor what he had seen. He spoke and sang to the bird, telling her stories of ancient times and wise wisdom. To his delight, the bird listened intently, laughing sometimes and asking questions.
#So, every day the wise owl and the visitor bird met in the forest, sharing stories and wisdom. Merlin learned that even the most ancient of birds could carry the light of the universe on their wings.
#One autumn, while the owl was resting in his nest, he heard a strange noise rustling the leaves. To his surprise, the noise was the song of a young migrating bird. The bird was going to leave his nest to find a new home in a distant land.
#Merlin felt a pang of sadness. He knew that soon the young bird would face the cold of the winter and the harshness of the journey. He thought of a special nest made of frost, sheltered by tiny ice crystals, which would keep the little bird warm during its journey.
#One day, the young bird arrived at the safe nest.<end_of_turn>



inputs = tokenizer.encode(txt,
                          return_tensors="pt",
                          add_special_tokens=False).to("cuda:0")
max_new_tokens=700
generation_config = GenerationConfig(
              max_new_tokens=max_new_tokens,
              temperature=0.32,
              #top_p=0.9,
              top_k=45,
              repetition_penalty=1.,  #1.1
              do_sample=True,
          )
outputs = model.generate(generation_config=generation_config,
                         input_ids=inputs,
                         stopping_criteria=stopping_criteria_list,)
tokenizer.decode(outputs[0], skip_special_tokens=False) #True