Xenova/multilingual-e5-small · Missing the following inputs: token_type

Nov 18, 2023

Apparently the token_type_ids are missing. The intfloat repo is also not working with transformers.js.

models.js:351 Uncaught (in promise) Error: An error occurred during model execution: "Missing the following inputs: token_type_ids.
    at validateInputs (models.js:351:15)
    at sessionRun (models.js:379:33)
    at Function.encoderForward [as _forward] (models.js:643:18)
    at Function.forward (models.js:929:27)
    at Function._call (models.js:918:27)
    at Function.closure [as model] (core.js:77:28)
    at Function._call (pipelines.js:119:34)
    at Function._call (pipelines.js:906:45)
    at closure (core.js:77:28)
    at embed (worker.js:44:22)

Xenova

Owner Nov 18, 2023

Hi there 👋 This is a known problem with the model, due to the fact that the model author decided to mix the type of tokenizer and the model. See here for a workaround: https://github.com/xenova/transformers.js/issues/324

do-me

Nov 18, 2023

Thanks a lot for linking the issue! Maybe it'd be worth mentioning it in the model card too?

Xenova

Owner Nov 19, 2023

To avoid issues like this in future, it is probably best to just account for this in the library. It should be as simple as adding a line like this to here, to add default input token ids if they are not present. Would you by any chance be interested in making a PR for this?

do-me

Nov 19, 2023

Sure, great idea. Will do so today!

do-me

Nov 19, 2023

Fixed it here, tested and works now.

do-me

Nov 20, 2023

The fix is merged in transformers.js, see https://github.com/xenova/transformers.js/pull/403#event-11004783800.
It should work flawlessly now.

do-me changed discussion status to closed Nov 20, 2023