What is the actual context size of mistralai/Mixtral-8x7B-Instruct-v0.1 model
#186
by
Pradeep1995
- opened
What is the actual context size of mistralai/Mixtral-8x7B-Instruct-v0.1 model
Context size as in context window? on Mistral AI's website you can read about the AI model. But it has a context window of 32K.
Actually when running inference I get this error. So I guess it's not 32k...
@Pradeep1995
@CR33STL
Have you saved the tokenizer? If you set the "model_max_length" to 1024 and your input length is now larger than 1024, it would generate this warning.
But if you are using the original tokenizer, the length is somehow "unlimited" as shown here
During their training, the context length used was 32768, as shown in the paper.