---
language:
- sv
license: llama3.1
library_name: transformers
tags:
- unsloth
datasets:
- neph1/bellman-7b-finetune
- neph1/codefeedback-swedish
---

# Model Card for Bellman

This version of bellman is finetuned from llama-3.1-instruct-8b.
It's finetuned for prompt question answering, based on a dataset created from 
Swedish wikipedia, with a lot of Sweden-centric questions.
New from previous versions is questions from a translated code-feedback dataset, as well as a number of stories. It's not great at generating stories, 
but better than previosly.

Try out the Q8 version here: https://huggingface.co/spaces/neph1/bellman (cpu)

![image/png](https://cdn-uploads.huggingface.co/production/uploads/653cd3049107029eb004f968/IDGX3d9lGe6yx-yHjsrav.png)

## Model Details

Training run on 240724:

Step 	Training Loss 	Validation Loss<br>
25 	    1.352200 	1.034565<br>
50 	    1.033600 	1.009348<br>
75 	    1.022400 	0.996665<br>
100 	1.002900 	0.988050<br>
125 	1.014600 	0.981633<br>
150 	1.006300 	0.975584<br>
175 	0.988800 	0.970966<br>
200 	0.985300 	0.967037<br>
225 	0.992400 	0.964120<br>
250 	0.950000 	0.962472<br>
275 	0.931000 	0.960848<br>
300 	0.932000 	0.958946 <-- picked checkpoint <br>

### Training Parameters
  per_device_train_batch_size = 4,<br>
  gradient_accumulation_steps = 16,<br>
  num_train_epochs=3,<br>
  warmup_steps = 5,<br>
  learning_rate = 3e-5,<br>
  logging_steps = 25,<br>
  optim = "adamw_8bit",<br>
  weight_decay = 0.01,<br>
  lr_scheduler_type = "linear",<br>
  seed = 3407,<br>
  per_device_eval_batch_size = 2,<br>
  eval_strategy="steps",<br>
  eval_accumulation_steps = 32,<br>
  eval_steps = 25,<br>
  eval_delay = 0,<br>
  save_strategy="steps",<br>
  save_steps=50,<br>
  
### Model Description


- **Developed by:** Me
- **Funded by:** Me
- **Model type:** Instruct
- **Language(s) (NLP):** Swedish
- **License:** llama-3.1
- **Finetuned from model:** Llama3.1 Instruct 8b

## Model Card Contact

rickard@mindemia.com