---
language:
- en
license: apache-2.0
tags:
- text-generation-inference
- transformers
- unsloth
- llama
- trl
- sft
---

<style>
  img{
    border-radius: 1rem;
  }
@import url('https://fonts.googleapis.com/css2?family=Vollkorn:ital,wght@0,400..900;1,400..900&display=swap');
</style>

<div style="background-color: transparent; border-radius: .5rem; padding: 2rem; font-family: monospace; font-size: .85rem; text-align: justify;">
  
![palmer-004](https://huggingface.co/appvoid/palmer-004-original/resolve/main/palmer-004.jpeg)

  
#### palmer turbo

This model has a slightly different architecture and training style:

1. The model was followed by a continual pretraining (lm_head + embedding layers were tuned).
2. Base model was pretrained on 75k instruction/response pairs and merged.
3. Similar architecture than palmer series but smaller in context size (8192)

In short, palmer is now half the size, twice the speed and almost same overall performance with a notable improvement on mmlu and arc challenge instead of winogrande. As of Wed 17 Jul, it beats all models =< 0.5b on hellaswag.

As all palmer models, the model is biased to respond to answers without using any specific prompt, feel free to further fine-tune it for your specific use case.

#### benchmarks

These are zero-shot evaluations performed on current state-of-the-art language models.

| Model                          | MMLU  | ARC-C | HellaSwag | PIQA   | Winogrande | Average |
|--------------------------------|-------|-------|-----------|--------|------------|---------|
| smollm-360m                    | 0.2537|**0.3626**| 0.5350    | 0.7116 | 0.5659     | 0.4858  |
| tinyllama                      | 0.2577| 0.3029| 0.5935    | 0.7329 | 0.5959     | 0.4966  |
| qwen2-0.5b                     |**0.4413**| 0.2892| 0.4905    | 0.6931 | 0.5699  | 0.4968  |
| danube3-500m-chat (current sota)| 0.2554|**0.3626**|0.6072 | 0.7432 | 0.6140     | 0.5164  |
| palmer-004-turbo               |0.2736|0.3558|**0.6179**|0.7367     | 0.6117     |**0.5191**|
| palmer-004                     | 0.2661| 0.3490| 0.6173  |**0.7481**|**0.6417**  |**0.5244**|


#### thanks to

- h2oai: performant base model provider
- teknium: openhermes dataset provider
- unsloth: tooling for training software
</div>