nzi_latn_5mb / added_tokens.json
goldfish-models's picture
Upload nzi_latn_5mb tokenizer.
5afad59
raw
history blame
975 Bytes
{"[XXXXX21]": 20455, "[XXXXX39]": 20473, "[XXXXX45]": 20479, "[XXXXX31]": 20465, "[XXXXX22]": 20456, "<pad>": 20432, "[XXXXX0]": 20434, "[XXXXX13]": 20447, "[XXXXX18]": 20452, "[XXXXX38]": 20472, "[XXXXX41]": 20475, "[XXXXX24]": 20458, "[XXXXX17]": 20451, "[SEP]": 20431, "[XXXXX33]": 20467, "[XXXXX14]": 20448, "[XXXXX43]": 20477, "[XXXXX16]": 20450, "[XXXXX10]": 20444, "[XXXXX44]": 20478, "[XXXXX40]": 20474, "[XXXXX9]": 20443, "[MASK]": 20433, "[XXXXX6]": 20440, "[XXXXX42]": 20476, "[XXXXX30]": 20464, "[XXXXX8]": 20442, "[XXXXX3]": 20437, "[XXXXX28]": 20462, "[XXXXX7]": 20441, "[XXXXX5]": 20439, "[XXXXX2]": 20436, "[XXXXX12]": 20446, "[XXXXX26]": 20460, "[XXXXX29]": 20463, "[XXXXX11]": 20445, "[XXXXX27]": 20461, "[XXXXX35]": 20469, "[XXXXX34]": 20468, "[XXXXX19]": 20453, "[XXXXX15]": 20449, "[XXXXX23]": 20457, "[XXXXX36]": 20470, "[XXXXX1]": 20435, "[CLS]": 20430, "[XXXXX4]": 20438, "[XXXXX25]": 20459, "[XXXXX32]": 20466, "[XXXXX20]": 20454, "[XXXXX37]": 20471}