openflamingo/OpenFlamingo-9B-vitl-mpt7b · Why the performance on VQAv2 is lower compared to the results reported in the openflamingo paper?

Hi there,

thanks for your work!

The performance on VQAv2 of this 9B model is 50.3, 50.5, 52.8, 52.3 and 50.5 from 0 to 32 shots. However, in the openflamingo paper Table 4, the 0, 4, 32 shot got 52.7, 54.8, and 53.3 which are better than 50.3, 50.5 and 50.5 respectively. The same applies to OK-VQA.
My reproduction presents 51.28, 53.43 and 53.5 given 0, 4, 8 shots respectively.

Question 1, why there is such differences between scores in this model card and the reported scores in the OF paper?

Question 2, I cannot reproduce the score compared to the paper, any suggestions?

Best regards.