---
base_model: Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-1.0
library_name: transformers
pipeline_tag: text-generation
datasets:
- meta-math/MetaMathQA
---

<a href="https://ibb.co/4jPnqYk"><img src="https://i.ibb.co/qRk3SW6/DALL-E-2024-08-08-05-52-48-Craft-an-epic-and-historic-image-for-a-model-card-blending-elements-of-an.webp" alt="DALL-E-2024-08-08-05-52-48-Craft-an-epic-and-historic-image-for-a-model-card-blending-elements-of-an" border="0"></a>

# Model Card for Na0s/Llama-3.1-8B-Pruned-4-Layers_LoRA-PEFT-2.0

## Model Details

### Model Description
- **Finetuned from model:[Na0s/Llama-3.1-8b-Pruned-4-Layers-1.0]** 

<!-- Provide a quick summary of what the model is/does. -->

## Training Details

    model = FastLanguageModel.get_peft_model(
    model,
    r = 4, 
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = 4,
    lora_dropout = 0.05, 
    bias = "none",    
   
    use_gradient_checkpointing = "unsloth", 
    random_state = 3407,
    use_rslora = False,  
    loftq_config = None, 
    )

    from trl import SFTTrainer
    from transformers import TrainingArguments
    from unsloth import is_bfloat16_supported

    trainer = SFTTrainer(
    model = model,
    tokenizer = tokenizer,
    train_dataset = dataset,
    dataset_text_field = "completion",
    max_seq_length = max_seq_length,
    dataset_num_proc = 2,
    packing = False, 
    args = TrainingArguments(
        per_device_train_batch_size = 10,
        gradient_accumulation_steps = 4,
        warmup_steps = 5,
        max_steps=5000,
        learning_rate = 2e-4,
        fp16 = not is_bfloat16_supported(),
        bf16 = is_bfloat16_supported(),
        logging_steps = 1,
        optim = "adamw_8bit",
        weight_decay = 0.01,
        lr_scheduler_type = "cosine",
        seed = 3407,
        output_dir = "outputs_4",
        push_to_hub=True,
        hub_always_push=True,
    ),
    )



### Training Data

[meta-math/MetaMathQA]

## Evaluation

MMLU Pro 0-shot: 0.2872


#### Evaluation Data

<!-- This should link to a Dataset Card if possible. -->

[TIGER-AI-Lab/MMLU-Pro]



## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).