stanfordnlp

/

stanza-ar

like 0

Follow

Stanford NLP 87

I tried using this tokenizer on diacritized texts but it fails to recognize the SHADDA (◌ّ) and replaces it with token. It also fails to tokenize MultiWordTokens correctly when the input text is diacritized (Tried with input "وَإنَّما،")