glk_arab_5mb / added_tokens.json
goldfish-models's picture
Upload glk_arab_5mb tokenizer.
b6d74f1
raw
history blame
275 Bytes
{"[CLS]": 38897, "[XXXXX3]": 38904, "[XXXXX1]": 38902, "[XXXXX4]": 38905, "[XXXXX9]": 38910, "<pad>": 38899, "[XXXXX0]": 38901, "[XXXXX8]": 38909, "[XXXXX5]": 38906, "[XXXXX7]": 38908, "[MASK]": 38900, "[XXXXX2]": 38903, "[XXXXX6]": 38907, "[XXXXX10]": 38911, "[SEP]": 38898}