---
license: cc-by-nc-4.0
language:
- pl
tags:
- llama
- qlora
- instruct
---

This repo contains a qlora adapter for [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf), trained on 1B tokens (available [here](https://huggingface.co/Azurro/llama-2-7b-qlora-polish)) and subsequently fine-tuned on a private instructions dataset, exclusively in Polish.

The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters:

* context length: 2048
* batch_size: 16
* learning_rate: 0.0001
* lora_r: 64
* lora_alpha: 16
* lora_modules: all
* lora_dropout: 0.0
* weight_decay: 0.1
* max_grad_norm: 0.3
* double_quant, nf4
* optimizer: paged_adamw_32bit (beta2: 0.999)

This adapter allows the model to speak Polish more accurately than vanilla [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf).

<p align="center">
  <img src="https://huggingface.co/Azurro/llama-2-7b-qlora-polish-instruct/raw/main/llama-2-7b-qlora-polish-instruct.jpg">
</p>