README.md

metadata

license: mit
tags:
  - tokenizers
  - Tokenizers.js
  - Transformers
  - experimental
  - auto tokenizer
  - llama
  - llama 2
  - custom code

Experimental tokenizer

• Derived from the LLaMa-2 tokenizer