metadata

license: apache-2.0
language:
  - ko
  - en
pipeline_tag: text-generation
tags:
  - arxiv:2405.15640
  - gecko
  - pretrained
library_name: transformers

GECKO: Generative Language Model for English, Code and Korean

Paper | GitHub |

GECKO-7B

GECKO is a 7B parameter deconder-only transformer pretrained on Korean, English and code. It is trained on 200 billion tokens and use terabytes of Korean corpus. GECKO is an open-source model released under Apache 2.0 License. For more details about our model, please read our technical report.

Model Details

GECKO is a generative language model using Llama architecture. Therefore, our model is easlily integrated with other frameworks which support Llama.

	Training Data	Params	Content Length	GQA	Tokens	LR
GECKO	A mix of publicly available online data	7B	8k	X	200B	3.0 x 10-4

Usage

~14GB RAM is the required minimum memory size with half-precision like float16 or bfloat16.

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = 'kifai/GECKO-7B'
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.bfloat16, device_map="auto")

text = """이 HTML 코드가 어떤 기능을 하는지 설명하고, 그 설명을 영어로 제공해주세요.
\```html
<button onclick="alert('Welcome!')">Click Me</button>
\```
"""
inputs = tokenizer(text, return_tensors='pt')['input_ids'].to('cuda')
output = model.generate(inputs, max_new_tokens=512, repetition_penalty=1.2)
print(tokenizer.decode(output[0], skip_special_tokens=True))
# 이 HTML 코드가 어떤 기능을 하는지 설명하고, 그 설명을 영어로 제공해주세요.
# \```html
# <button onclick="alert('Welcome!')">Click Me</button>
# \```
# 
# ## Description
# 
# This is a button that will display the message "Welcome!" when clicked.
# 
# ## Expected Output
# 
# The expected output should be:
# 
# \```text
# Welcome!
# \```

Limitation

GECKO is a generative language model that comes with some risks. Its testing has mainly been conducted in Korean and has not covered all possible scenarios. As with all large language models, the outputs from GECKO cannot be predicted in advance and might somtimes be inaccurate, biased, or otherwise problematic. Therefore, developers should conduct safety testing and fine-tune model for the intended uses before deploying it.

License

GECKO is released under Apache 2.0 license.

Citation

@misc{oh2024gecko,
      title={GECKO: Generative Language Model for English, Code and Korean}, 
      author={Sungwoo Oh and Donggyu Kim},
      year={2024},
      eprint={2405.15640},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Acknowledgement

The training is supported by TPU Research Cloud program.

Contact

We look forward to hearing you and collaborating with us

Sungwoo Oh [LinkedIn]
Donggyu Kim