chrisociepa's picture
Update README.md
004188b
|
raw
history blame
933 Bytes
metadata
license: apache-2.0
language:
  - pl
tags:
  - qlora
  - instruct

This repo contains a qlora adapter for Mistral-7B-Instruct-v0.1, fine-tuned on a private instructions dataset, exclusively in Polish.

The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters:

  • context length: 2048
  • batch_size: 16
  • learning_rate: 0.0001
  • lora_r: 64
  • lora_alpha: 16
  • lora_modules: all
  • lora_dropout: 0.0
  • weight_decay: 0.1
  • max_grad_norm: 0.3
  • double_quant, nf4
  • optimizer: paged_adamw_32bit (beta2: 0.999)

During the training of the adapter, the instruction format was maintained. Each prompt should be surrounded by [INST] and [\INST] tokens to leverage instruction fine-tuning.

This adapter allows the model to speak Polish more accurately than vanilla Mistral-7B-Instruct-v0.1.