BigBird code availability
#5
by
logvinata
- opened
Hi!
The BigBird model card says that the code is available on GitHub, however, it is not. Will you make it public?
Also could you please share what resources were used to pre-train both GenaLM models (hardware and time)?
Hi! We use the BigBird model code from HuggingFace Transformers library, e.g.:
from transformers import AutoTokenizer, BigBirdForMaskedLM
tokenizer = AutoTokenizer.from_pretrained('AIRI-Institute/gena-lm-bigbird-base-t2t')
model = BigBirdForMaskedLM.from_pretrained('AIRI-Institute/gena-lm-bigbird-base-t2t')
Both models were trained for about 1 month on 8 - 16 A100. We are working on a publication about GENA-LM models and will update the code uploaded to GitHub with more details about the pre-training.
Great! Thank you!