license: apache-2.0
base_model: upstage/SOLAR-10.7B-v1.0
tags:
- generated_from_trainer
model-index:
- name: yanolja/KoSOLAR-10.7B-v0.1
results: []
Discord
If you're passionate about the field of Large Language Models and wish to exchange knowledge and insights, we warmly invite you to join our Discord server. It's worth noting that Korean is the primary language used in this server. The landscape of LLM is evolving rapidly, and without active sharing, our collective knowledge risks becoming outdated swiftly. Let's collaborate and drive greater impact together! Join us here: https://discord.gg/b27bAHg95m.
Caution
This model is DEPRECATED due to an issue with the tokenizer. A new, corrected version will be uploaded shortly. We strongly advise against fine-tuning this model until the updated version is available. Details for the new version will be provided in a separate model card.
yanolja/KoSOLAR-10.7B-v0.1
This model is a Korean vocabulary-extended version of upstage/SOLAR-10.7B-v1.0, specifically pre-trained on various Korean web-crawled datasets available on HuggingFace. Our approach was to expand the model's understanding of Korean by pre-training the embeddings for new tokens while preserving the original parameters of the base model.
Model Description
Most parameters of upstage/SOLAR-10.7B-v1.0 were kept frozen during our training process. Only the embeddings for the newly added Korean tokens in the embed_tokens layer and the lm_head layer were pre-trained. This approach allowed us to enhance the model's performance in Korean while maintaining its original English capabilities.
Intended Uses & Limitations
No instruction tuning has been performed on this model. We recommend further training for specific purposes with caution, as it was primarily enhanced for Korean language understanding.
Training and Evaluation Data
The model was pre-trained on various Korean web-crawled datasets openly available on HuggingFace.
Training Procedure
Clarification on "Pre-trained"
It's essential to understand what "pre-trained" means in the context of this model. While the base model was already pre-trained on a broad, non-task-specific corpus of data, we further pre-trained only the embeddings for the expanded Korean vocabulary. This means that we did not alter the other existing parameters from the base model at all. This approach ensures a robust understanding of both English and Korean.
Training Hyperparameters
The following hyperparameters were used during training:
- learning_rate: 0.0003
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- num_devices: 8
- gradient_accumulation_steps: 4
- total_train_batch_size: 256
- total_eval_batch_size: 64
- optimizer: Adam with betas=(0.9, 0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_steps: 10
- num_epochs: 1
Training Results
upstage/SOLAR-10.7B-v1.0
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| kmmlu | N/A | none | 0 | acc | 0.3004 | ± | 0.0528 |
| gsm8k | Yaml | get-answer | 5 | exact_match | 0.5625 | ± | 0.0137 |
| hellaswag | Yaml | none | 0 | acc | 0.6393 | ± | 0.0048 |
| mmlu | N/A | none | 0 | acc | 0.6305 | ± | 0.1452 |
| truthfulqa | N/A | none | 0 | acc | 0.4096 | ± | 0.0467 |
| winogrande | Yaml | none | 0 | acc | 0.7443 | ± | 0.0123 |
yanolja/KoSOLAR-10.7B-v0.1
| Groups | Version | Filter | n-shot | Metric | Value | Stderr | |
|---|---|---|---|---|---|---|---|
| kmmlu | N/A | none | 0 | acc | 0.2948 | ± | 0.0537 |
| gsm8k | Yaml | get-answer | 5 | exact_match | 0.5527 | ± | 0.0137 |
| hellaswag | Yaml | none | 0 | acc | 0.6392 | ± | 0.0048 |
| mmlu | N/A | none | 0 | acc | 0.6303 | ± | 0.1411 |
| truthfulqa | N/A | none | 0 | acc | 0.3618 | ± | 0.0472 |
| winogrande | Yaml | none | 0 | acc | 0.7459 | ± | 0.0122 |
Framework Versions
- Transformers 4.37.0.dev0
- Pytorch 2.1.2+cu121
- Datasets 2.16.0
- Tokenizers 0.15.0