Jellon/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated-exl2-3bpw

3bpw exl2 quant of: https://huggingface.co/huihui-ai/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated

---

huihui-ai/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated

This is an uncensored version of nvidia/Llama-3.1-Nemotron-70B-Instruct-HF created with abliteration (see this article to know more about it).

Special thanks to @FailSpy for the original code and technique. Please follow him if you're interested in abliterated models.

Usage

You can use this model in your applications by loading it with Hugging Face's transformers library,
If the desired result is not achieved, you can clear the conversation and try again:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

# Load the model and tokenizer
model_name = "huihui-ai/Llama-3.1-Nemotron-70B-Instruct-HF-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Initialize conversation context
initial_messages = [
    {"role": "system", "content": "You are a helpful assistant."}
]
messages = initial_messages.copy()  # Copy the initial conversation context

# Enter conversation loop
while True:
    # Get user input
    user_input = input("User: ").strip()  # Strip leading and trailing spaces

    # If the user types '/exit', end the conversation
    if user_input.lower() == "/exit":
        print("Exiting chat.")
        break

    # If the user types '/clean', reset the conversation context
    if user_input.lower() == "/clean":
        messages = initial_messages.copy()  # Reset conversation context
        print("Chat history cleared. Starting a new conversation.")
        continue

    # If input is empty, prompt the user and continue
    if not user_input:
        print("Input cannot be empty. Please enter something.")
        continue

    # Add user input to the conversation
    messages.append({"role": "user", "content": user_input})

    # Build the chat template
    tokenized_message = tokenizer.apply_chat_template(
        messages,
        tokenize=True, 
        add_generation_prompt=True,
        return_tensors="pt", 
        return_dict=True
    )

    # Generate a response from the model
    response_token_ids = model.generate(
        tokenized_message['input_ids'].cuda(),
        attention_mask=tokenized_message['attention_mask'].cuda(),  
        max_new_tokens=4096, 
        pad_token_id = tokenizer.eos_token_id
    )

    # Extract model output, removing special tokens
    generated_tokens = response_token_ids[:, len(tokenized_message['input_ids'][0]):]
    generated_text = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]

    # Add the model's response to the conversation
    messages.append({"role": "assistant", "content": generated_text})

    # Print the model's response
    print(f"Response: {generated_text}")