Update contamination_report.csv
What are you reporting:
Contaminated model(s): GPT-4
Contaminated corpora:
conll2003
nyu-mll/glue
rajpurkar/squad_v2
https://catalog.ldc.upenn.edu/LDC2006T06
quac;;GPT-4;Model
natural_questions
google/boolq
Contaminated split(s): If the dataset has Train, Development and/or Test splits please report the contaminated split(s). You can report a percentage of the dataset contaminated; if the entire dataset is compromised, report 100%.
It is unclear what is the percentage, we just know the model regurgitates training validataion and test data and or metadata of each.
You may also report instances where there is no contamination. In such cases, follow the previous instructions but report a contamination level of 0%.
Briefly describe your method to detect data contamination
- Model-based approach
Description of your method, 3-4 sentences. Evidence of data contamination (Read below):
Prompt GPT and see that it knows to return metadata and training and val\test examples on its own.
see more here
https://hitz-zentroa.github.io/lm-contamination/blog/
Data-based approaches
Data-based approaches identify evidence of data contamination in a pre-training corpus by directly examining the dataset for instances of the evaluation data. This method involves algorithmically searching through a large pre-training dataset to find occurrences of the evaluation data. You should provide evidence of data contamination in the form: "dataset X appears in line N of corpus Y," "dataset X appears N times in corpus Y," or "N examples from dataset X appear in corpus Y."
Model-based approaches
Model-based approaches, on the other hand, utilize heuristic algorithms to infer the presence of data contamination in a pre-trained model. These methods do not directly analyze the data but instead assess the model's behavior to predict data contamination. Examples include prompting the model to reproduce elements of an evaluation dataset to demonstrate memorization (i.e https://hitz-zentroa.github.io/lm-contamination/blog/) or using perplexity measures to estimate data contamination (). You should provide evidence of data contamination in the form of evaluation results of the algorithm from research papers, screenshots of model outputs that demonstrate memorization of a pre-training dataset, or any other form of evaluation that substantiates the method's effectiveness in detecting data contamination. You can provide a confidence score in your predictions.
Citation
Is there a paper that reports the data contamination or describes the method used to detect data contamination?
Blog post not paper, so we can create a bib if we want
URL: [https://aclanthology.org/2023.findings-emnlp.722/](https://hitz-zentroa.github.io/lm-contamination/blog/)
Citation: @inproceedings{...
Important! If you wish to be listed as an author in the final report, please complete this information for all the authors of this Pull Request.
- Full name: Leshem Choshen
- Institution: MIT-IBM watson AI lab, MIT
- Email: [email protected]
Thank you for the PR @borgr !
Could you make two small changes in your PR? These experiments were conducted using GPT-3.5, not GPT-4. Also, could you set the contamination percentage for the contaminated splits (Train Split; Development Split; Test Split) to 100% and 0% for the non-contaminated ones? Although we don't know the exact percentage of the corpus that has been compromised, it will be indicated in the table that the model is contaminated.
Hmm... Embarrassing but I don't find how to do it. I see how can I edit the title or text, but not the commit itself. Neither do I find ainfo on how to do it not through the web API. I can upload a new PR...
Other comments, why ";" and not tab or "," like normal fomats use (tsv\csv) so tools can easily edit and save?
Created a new PR instead for now
Hi @borgr !
Directly from HuggingFace is a bit tricky, you need to click on the branch name from: refs/pr/4 and then edit it there.
If you work from local you can do the following steps (see documentation):
git fetch origin refs/pr/4:pr/4
git checkout pr/4
# Do your changes
git add .
git commit -m "Add your change"
git push origin pr/4:refs/pr/4
I will close this PR so we can continue in the new one :)