microsoft

/

Florence-2-large

like 1.2k

Follow

Microsoft 4,746

I am trying to fine-tune CAPTIONS because the trained dataset isn't representative of the images I'm working with.
This is how I'm loading the model

def load_model():
    model_id = 'microsoft/Florence-2-large'
    revision = 'aec3bc57822662953317bc228d279bad98b7819c'#'refs/pr/6'
    model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, revision=revision).to(cfg.device)
    processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True, revision=revision)
    return model, processor

And I copied the code from: https://huggingface.co/blog/finetune-florence2 and https://colab.research.google.com/github/roboflow-ai/notebooks/blob/main/notebooks/how-to-finetune-florence-2-on-detection-dataset.ipynb?ref=blog.roboflow.com#scrollTo=bC06Mc7jOdpY to load the data. The issue I'm having is in the training script running the forward pass

            input_ids = inputs["input_ids"]
            pixel_values = inputs["pixel_values"]
            labels = processor.tokenizer(
                text=answers,
                return_tensors="pt",
                padding=True,
                return_token_type_ids=False
            ).input_ids.to(cfg.device)

            outputs = model(input_ids=input_ids, pixel_values=pixel_values, labels=labels)

  File "cache/models/modules/transformers_modules/microsoft/Florence-2-large/aec3bc57822662953317bc228d279bad98b7819c/modeling_florence2.py", line 2759, in forward
    return Florence2Seq2SeqLMOutput(
TypeError: __init__() got an unexpected keyword argument 'loss

I checked the file and the error is correct but when I add it there is another unexpected keyword arguement