---
library_name: transformers
tags:
- mergekit
- merge
pipeline_tag: text-generation
---

<a href="https://ibb.co/k9z9yyz"><img src="https://i.ibb.co/61G1ZZG/DALL-E-2024-08-08-05-39-49-Create-a-visually-captivating-image-for-a-model-card-representing-a-prune.webp" alt="DALL-E-2024-08-08-05-39-49-Create-a-visually-captivating-image-for-a-model-card-representing-a-prune" border="0"></a><br /><a target='_blank' href='https://usefulwebtool.com/fr/convertir-minuscules-majuscules'></a><br />

# Na0s/Llama-3.1-8b-Pruned-4-Layers

This is a merge of meta-llama/Meta-Llama-3.1-8B created using [mergekit](https://github.com/cg123/mergekit), with respect to the paper ["The Unreasonable Ineffectiveness of the Deeper Layers"](https://arxiv.org/pdf/2403.17887)


## Merge Details
### Merge Method

This model was merged using the passthrough merge method.

### Models Merged

The following models were included in the merge:
* [meta-llama/Meta-Llama-3.1-8B](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B)

### Configuration

The following YAML configuration was used to produce this model:

```yaml
dtype: bfloat16
merge_method: passthrough
slices:
- sources:
  - layer_range: [0, 23]
    model: meta-llama/Meta-Llama-3.1-8B
- sources:
  - layer_range: [28, 32]
    model: meta-llama/Meta-Llama-3.1-8B
```

## Evaluation

MMLU Pro 0-shot: 0.2642


#### Evaluation Data

<!-- This should link to a Dataset Card if possible. -->

[TIGER-AI-Lab/MMLU-Pro]


## Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->