RichardErkhov/spow12_-_POLAR-14B_4.3_very_big_sft-gguf

Quantization made by Richard Erkhov.

Github

Discord

Request more models

POLAR-14B_4.3_very_big_sft - GGUF

• Model creator: https://huggingface.co/spow12/
• Original model: https://huggingface.co/spow12/POLAR-14B_4.3_very_big_sft/

Name	Quant method	Size
POLAR-14B_4.3_very_big_sft.Q2_K.gguf	Q2_K	4.92GB
POLAR-14B_4.3_very_big_sft.IQ3_XS.gguf	IQ3_XS	5.47GB
POLAR-14B_4.3_very_big_sft.IQ3_S.gguf	IQ3_S	5.77GB
POLAR-14B_4.3_very_big_sft.Q3_K_S.gguf	Q3_K_S	5.74GB
POLAR-14B_4.3_very_big_sft.IQ3_M.gguf	IQ3_M	5.97GB
POLAR-14B_4.3_very_big_sft.Q3_K.gguf	Q3_K	6.4GB
POLAR-14B_4.3_very_big_sft.Q3_K_M.gguf	Q3_K_M	6.4GB
POLAR-14B_4.3_very_big_sft.Q3_K_L.gguf	Q3_K_L	6.97GB
POLAR-14B_4.3_very_big_sft.IQ4_XS.gguf	IQ4_XS	7.18GB
POLAR-14B_4.3_very_big_sft.Q4_0.gguf	Q4_0	7.48GB
POLAR-14B_4.3_very_big_sft.IQ4_NL.gguf	IQ4_NL	7.57GB
POLAR-14B_4.3_very_big_sft.Q4_K_S.gguf	Q4_K_S	7.54GB
POLAR-14B_4.3_very_big_sft.Q4_K.gguf	Q4_K	7.97GB
POLAR-14B_4.3_very_big_sft.Q4_K_M.gguf	Q4_K_M	7.97GB
POLAR-14B_4.3_very_big_sft.Q4_1.gguf	Q4_1	8.3GB
POLAR-14B_4.3_very_big_sft.Q5_0.gguf	Q5_0	9.12GB
POLAR-14B_4.3_very_big_sft.Q5_K_S.gguf	Q5_K_S	9.12GB
POLAR-14B_4.3_very_big_sft.Q5_K.gguf	Q5_K	9.37GB
POLAR-14B_4.3_very_big_sft.Q5_K_M.gguf	Q5_K_M	9.37GB
POLAR-14B_4.3_very_big_sft.Q5_1.gguf	Q5_1	9.94GB
POLAR-14B_4.3_very_big_sft.Q6_K.gguf	Q6_K	10.87GB
POLAR-14B_4.3_very_big_sft.Q8_0.gguf	Q8_0	14.07GB

Original model description:

library_name: transformers license: cc-by-nc-4.0 language: - ko - en

spow12/POLAR-14B_4.3_very_big_sft

Model Description

This model is a Supervised fine-tuned version of x2bee/POLAR-14B-v0.2 with DeepSpeed and trl for korean.

Trained Data

• Trained with public data and private data and Generated data (about 50k)

Usage

from transformers import TextStreamer, pipeline, AutoTokenizer, AutoModelForCausalLM

model_id = 'spow12/POLAR-14B_4.3_very_big_sft'
tokenizer = AutoTokenizer.from_pretrained(model_id)
# %%
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2", 
    device_map='auto',
)
model.eval()

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, device_map='auto')

streamer = TextStreamer(tokenizer)

generation_configs = dict(
    max_new_tokens=2048,
    num_return_sequences=1, 
    temperature=0.1,
    # early_stopping=True,
    repetition_penalty=1.2,
    num_beams=1,
    do_sample=True,
    top_k=20,
    top_p=0.9,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.eos_token_id,
    streamer=streamer
)

sys_message = """당신은 친절한 챗봇으로서 상대방의 요청에 최대한 자세하고 친절하게 답해야합니다. 
사용자가 제공하는 정보를 세심하게 분석하여 사용자의 의도를 신속하게 파악하고 그에 따라 답변을 생성해야합니다.  

항상 매우 자연스러운 한국어로 응답하세요."""

message = [
    {
        'role': "system",
        'content': sys_message
    },
    {
        'role': 'user',
        'content': "현재의 경제상황에 대해 어떻게 생각해?."
    }
]
conversation = pipe(message, **generation_configs)
conversation[-1]

License

This model is licensed under the cc-by-nc-4.0. which allows others to share and adapt the model for non-commercial purposes.

Here is Original Readme.md