where to update max_prompt_len(to solve max_prompt_len <= params.max_seq_len, preferably using AWS JumpStart)
#121
by
wichofer
- opened
Maybe this is a silly and simple question but I cannot find where to do it in the documentation. Where should I update max_prompt_len?
For more context and use case: we deployed llama 3.1 8b instruct via AWS JumpStart, it works fine for very small prompts but with slightly bigger prompts it throws an error:
max_prompt_len <= params.max_seq_len
Solution seems simple: update those parameters, but I cannot find where to do it?
I have the additional need of updating that via the jumpstart wrapper, but that would be the next step, first I need to understand where it is updated in general.