|
--- |
|
inference: false |
|
language: |
|
- bg |
|
license: mit |
|
datasets: |
|
- oscar |
|
- chitanka |
|
- wikipedia |
|
tags: |
|
- torch |
|
--- |
|
|
|
# GPT-2 |
|
|
|
Pretrained model on Bulgarian language using a causal language modeling (CLM) objective. It was introduced in |
|
[this paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) |
|
and first released at [this page](https://openai.com/blog/better-language-models/). |
|
|
|
## Model description |
|
|
|
This is the **SMALL** version compressed via [progressive module replacing](https://arxiv.org/abs/2002.02925). |
|
|
|
The compression was executed on Bulgarian text from [OSCAR](https://oscar-corpus.com/post/oscar-2019/), [Chitanka](https://chitanka.info/) and [Wikipedia](https://bg.wikipedia.org/). |
|
|
|
## Intended uses & limitations |
|
|
|
You can use the raw model for: |
|
- text generation |
|
- auto-complete |
|
- spelling correction |
|
|
|
Or fine-tune it to a downstream task. |
|
|
|
### How to use |
|
|
|
Here is how to use this model in PyTorch: |
|
|
|
```python |
|
>>> from transformers import AutoModel, AutoTokenizer |
|
>>> |
|
>>> model_id = "rmihaylov/gpt2-small-theseus-bg" |
|
>>> tokenizer = AutoTokenizer.from_pretrained(model_id) |
|
>>> model = AutoModel.from_pretrained(model_id, trust_remote_code=True) |
|
>>> |
|
>>> input_ids = tokenizer.encode( |
|
>>> "Здравей,", |
|
>>> add_special_tokens=False, |
|
>>> return_tensors='pt') |
|
>>> |
|
>>> output_ids = model.generate( |
|
>>> input_ids, |
|
>>> do_sample=True, |
|
>>> max_length=50, |
|
>>> top_p=0.92, |
|
>>> pad_token_id=2, |
|
>>> top_k=0) |
|
>>> |
|
>>> output = tokenizer.decode(output_ids[0]) |
|
>>> |
|
>>> output = output.replace('<|endoftext|>', '\n\n\n') |
|
>>> output = output.replace('<|unknown|>', '') |
|
>>> output = output.replace('▁', ' ') |
|
>>> output = output.replace('<|n|>', '\n') |
|
>>> |
|
>>> print(output) |
|
|
|
Здравей, извинявай, но не мога да заспя. |
|
Джини се обърна и забеляза колко са прегърнати. |
|
— Почакай, Джини. Не мога да повярвам, че е възможно! Толкова искам да те видя. |
|
— Обеща |
|
``` |
|
|
|
### Limitations and bias |
|
|
|
As the openAI team themselves point out in their |
|
[model card](https://github.com/openai/gpt-2/blob/master/model_card.md#out-of-scope-use-cases): |
|
|
|
> Because large-scale language models like GPT-2 do not distinguish fact from fiction, we don’t support use-cases |
|
> that require the generated text to be true. |
|
> |
|
> Additionally, language models like GPT-2 reflect the biases inherent to the systems they were trained on, so we do |
|
> not recommend that they be deployed into systems that interact with humans > unless the deployers first carry out a |
|
> study of biases relevant to the intended use-case. We found no statistically significant difference in gender, race, |
|
> and religious bias probes between 774M and 1.5B, implying all versions of GPT-2 should be approached with similar |
|
> levels of caution around use cases that are sensitive to biases around human attributes. |
