File size: 2,102 Bytes
c5bff01 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
license: apache-2.0
language:
- en
---
## Empowering Character-level Text Infilling by Eliminating Sub-Tokens
<p align="center">
<a href="https://arxiv.org/abs/2405.17103">📄 Paper</a> •
<a href="https://github.com/SenseLLM/FIM-SE">🏠 Repo</a> •
<a href="https://huggingface.co/SenseLLM/FIM-SE-CL-13B">🤖 Models</a>
</p>
## Introduction
FIM-SE stands for Fill-In-the-Middle with both Starting and Ending character constraints. The proposed method addresses character-level infilling tasks by utilizing a line-level format to avoid predicting any sub-token in inference.
<hr>
## Models
| Model | Checkpoint | Size | License|
|:------|:-----------|:-----|:-------|
| FIM-SE-CL-7B | 🤗 [HF Link](https://huggingface.co/SenseLLM/FIM-SE-CL-7B) | 7B | [Llama2](https://ai.meta.com/llama/license/) |
| FIM-SE-CL-34B | 🤗 [HF Link](https://huggingface.co/SenseLLM/FIM-SE-CL-34B) | 13B | [Llama2](https://ai.meta.com/llama/license/) |
| FIM-SE-SC-1B | 🤗 [HF Link](https://huggingface.co/SenseLLM/FIM-SE-SC-1B) | 1B | [StarCoder](https://github.com/bigcode-project/starcoder/blob/main/LICENSE) |
| FIM-SE-SC-15B | 🤗 [HF Link](https://huggingface.co/SenseLLM/FIM-SE-SC-15B) | 15B | [StarCoder](https://github.com/bigcode-project/starcoder/blob/main/LICENSE) |
## How to Use
#### Prompt Format
As shown in the figure, the prompt is organized as
```text
<PRE>R-Prefix<SUF>R-Suffix<START>L-Prefix<END>F-Suffix<MID>
```
#### Inference Code
Please refer to our [GitHub Repo](https://github.com/SenseLLM/FIM-SE) for more technical details.
## Citation
If you find this repo useful for your research, please kindly cite our paper:
```
@misc{ren2024empowering,
title={Empowering Character-level Text Infilling by Eliminating Sub-Tokens},
author={Houxing Ren and Mingjie Zhan and Zhongyuan Wu and Hongsheng Li},
year={2024},
eprint={2405.17103},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
```
## Acknowledgments
We thank the following amazing projects that truly inspired us:
- [FIM](https://arxiv.org/abs/2207.14255) |