bizzare behavior?
Hi, just getting started using your Rebel model, but the performance is a bit strange... I'm getting certain completions, but in other cases it's hallucinating relationships and duplicating them. I found a good toy example, this is from a summary of the lord of the flies:
Up to this point the boys have maintained a fragile balance, with Jack’s willingness to enact violence offset by Ralph’s control of the means of lighting the fire and the symbolic power conferred by the conch. Once this balance is destroyed, and Jack controls both the means of sustaining the fire and keeping the boys obedient to his rule, Ralph is rendered powerless. Unlike Ralph, who expects the boys to be intrinsically motivated to work together, Jack is willing to exert external influence on boys who disobey him, and leads by force, rather than persuasion. Motivated by a fear of Jack’s violence as well as a mob mentality, the boys pursue Ralph across the island, even though he poses no actual threat. Even the twins Samneric, initially sympathetic to Ralph, give themselves over to Jack after he tortures them to reveal Ralph’s hiding place. The boys set a fire to flush Ralph out of the jungle, which signals a passing ship. The ship’s officer comes on shore, reintroducing civilization, and the boys realize
If I feed that into the model, I get:
Jack Samneric sibling Samneric Jack sibling Jack sibling Jack sibling Jack Samneric sibling Samneric sibling Samneric Jack sibling Samneric sibling Jack sibling Jack Samneric sibling Samneric sibling
Which is I guess one triplet, repeated several times. I haven't read your paper yet about optimising the chunk size that I feed the model, and if any preprocesing should be done... sorry if that's already explained!
This is expected as a shortcoming of autoregressive nature and inputing a text very different from those seen at training (paragraph vs. sentence), which leads to weird generations.
Try changing the input to separate sentences, and be aware that the model may still hallucinate, as pointed in other issues like:
https://github.com/Babelscape/rebel/issues/67
https://github.com/Babelscape/rebel/issues/17