Question regarding max_position_embeddings in config.json

by tarun360 - opened Sep 24

Sep 24

Hello,

For allenai/OLMo-7B-0724-Instruct-hf, in the model card its mentioned that the context lenght is 4096. Hence, max_position_embeddings in config.json should be set as 4096 right? Currently, its set as 2048.

tarun360

Sep 24

For allenai/OLMo-7B-0724-hf, its correctly set as 4096 here: https://huggingface.co/allenai/OLMo-7B-0724-hf/blob/main/config.json

natolambert

Ai2 org Sep 24

Good check! Fixed.

natolambert changed discussion status to closed Sep 24

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment