phantomcoder1996's picture
add tokenizer
af9869e
raw
history blame
385 Bytes
{"ء": 1, "آ": 2, "أ": 3, "ؤ": 4, "إ": 5, "ئ": 6, "ا": 7, "ب": 8, "ة": 9, "ت": 10, "ث": 11, "ج": 12, "ح": 13, "خ": 14, "د": 15, "ذ": 16, "ر": 17, "ز": 18, "س": 19, "ش": 20, "ص": 21, "ض": 22, "ط": 23, "ظ": 24, "ع": 25, "غ": 26, "ف": 27, "ق": 28, "ك": 29, "ل": 30, "م": 31, "ن": 32, "ه": 33, "و": 34, "ى": 35, "ي": 36, "|": 0, "[UNK]": 37, "[PAD]": 38}