BOS token in front of every text chunk

#22
by websterbei - opened

https://huggingface.co/microsoft/Phi-3.5-vision-instruct/blob/c881ad502f9acce6e9f713db2926b7f0328a28d8/processing_phi3_v.py#L414

tokenizer will automatically add BOS token to the front of each text chunk, just checking if this is the intended behavior? (i.e. how it was trained)

Sign up or log in to comment