Batch Inference
#17
by
dfrank
- opened
Hi I've been having some problems doing batch inference. The only thing that seems to work is using tokenizer.padding_side = "right" but the results I get are inconsistent with respect to a single inference (by a lot). Any advice?
Hi
@dfrank
, This is because padding the inputs to the right in batch inference can result in different token sequences than padding to the left during single inference. Make sure your model is set to evaluation mode (e.g., model.eval() in PyTorch). This will disable any dropout layers, which could introduce variability in your outputs. if you have any concerns let us know. Thank you.