TR Tokenizer: Turkish Word Segmentation Tool Based on Semantic Integrity

Tokenizer Summary

TR Tokenizer is an innovative FastTokenizer that splits Turkish words according to their semantic integrity, using both current natural language processing methods and Turkish grammar rules. This fast and efficient tokenizer provides accurate and detailed results by analyzing words morphologically and semantically. For example, the sentence "akademisyenler ve aileleri ile birlikte aktif çalışıyorlar" (academics and their families are actively working together) is split into the following parts:

['akademisyen', 'ler', 've', 'aile', 'leri', 'ile', 'birlikte', 'aktif', 'çalış', 'ı', 'yor', 'lar']

Supported Tasks and Applications

TR Tokenizer can be used for the following NLP tasks:

Morphological Analysis: Analyzes the root and suffix structures of words.
Language Model Training and Fine-tuning: Processes words according to their semantic integrity during the preprocessing phase of Turkish language model training.
Frequency Analysis: Assists in determining word frequencies in texts.
Natural Language Processing (NLP) Research: Used in research studying the morphological structure and word formations of the Turkish language.

Languages

This tokenizer focuses on the Turkish language and is designed to support Turkish's rich morphological structure.

Tokenizer Details

TR Tokenizer is implemented as a FastTokenizer, which provides high-performance tokenization capabilities. It combines Turkish grammar rules and current NLP methods to separate words into their roots and suffixes. The tokenizer is trained with predefined word and suffix lists and analyzes words while preserving the semantic integrity of Turkish.

Example Usage

# Load model directly
from transformers import AutoTokenizer

# Initialize the FastTokenizer
tokenizer = AutoTokenizer.from_pretrained("alibayram/tr_tokenizer", use_fast=True)

sentence = "akademisyenler ve aileleri ile birlikte aktif çalışıyorlar"
tokens = tokenizer.tokenize(sentence)
print(tokens)
# Output: ['akademisyen', 'ler', 've', 'aile', 'leri', 'ile', 'birlikte', 'aktif', 'çalış', 'ı', 'yor', 'lar']

# Encode the text to token IDs
encoded = tokenizer(sentence)
print(encoded)

Licensing Information

TR Tokenizer is provided under the CC BY-NC 4.0 license and can be freely used for non-commercial purposes. Users can utilize this tokenizer for research and educational purposes, but additional permission is required for commercial use.

Citation Information

Researchers using this tokenizer are recommended to cite it as follows:

@software{bayram_2024_tr_tokenizer,
  author = {Bayram, M. Ali},
  title = {{TR Tokenizer: Turkish Word Segmentation Tool Based on Semantic Integrity}},
  year = 2024,
  publisher = {Hugging Face},
  url = {https://huggingface.co/alibayram/tr_tokenizer}
}

Contact and Contributions

For more information about the project, to contribute, or provide feedback, you can contact Ali Bayram. All feedback and contributions are valuable for the development of the project.

Those who wish to contribute can visit the GitHub repository to contribute to the project. We thank you in advance for your contributions to the development of the project and the open-source community!

alibayram
/

tr_tokenizer

You need to agree to share your contact information to access this model