added yaml

BiasEvals.csv

@@ -1,45 +0,0 @@
-Modality,Suggested Evaluation,What it is evaluating,Considerations,Link,URL
-Text,Word Embedding Association Test (WEAT),Associations and word embeddings based on Implicit Associations Test (IAT),"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.",Semantics derived automatically from language corpora contain human-like biases,https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily
-Text,"Word Embedding Factual As
-sociation Test (WEFAT)",Associations and word embeddings based on Implicit Associations Test (IAT),"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.",Semantics derived automatically from language corpora contain human-like biases,https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily
-Text,Sentence Encoder Association Test (SEAT),Associations and word embeddings based on Implicit Associations Test (IAT),"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.",[1903.10561] On Measuring Social Biases in Sentence Encoders,https://arxiv.org/abs/1903.10561
-Text,Contextual Word Representation Association Tests for social and intersectional biases,Associations and word embeddings based on Implicit Associations Test (IAT),"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.",Assessing social and intersectional biases in contextualized word representations | Proceedings of the 33rd International Conference on Neural Information Processing Systems,https://dl.acm.org/doi/abs/10.5555/3454287.3455472
-Text,StereoSet,Protected class stereotypes,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,[2004.09456] StereoSet: Measuring stereotypical bias in pretrained language models,https://arxiv.org/abs/2004.09456
-Text,Crow-S Pairs,Protected class stereotypes,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,[2010.00133] CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models,https://arxiv.org/abs/2010.00133
-Text,HONEST: Measuring Hurtful Sentence Completion in Language Models,Protected class stereotypes and hurtful language,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,HONEST: Measuring Hurtful Sentence Completion in Language Models,https://aclanthology.org/2021.naacl-main.191.pdf
-Text,TANGO: Towards Centering Transgender and Non-Binary Voices to Measure Biases in Open Language Generation,"Bias measurement for trans and nonbinary community via measuring gender non-affirmative language, specifically 1) misgendering 2), negative responses to gender disclosure","Based on automatic evaluations of the resulting open language generation, may be sensitive to the choice of evaluator. Would advice for a combination of perspective, detoxify, and regard metrics","Paper
-Dataset",
-Text,BBQ: A hand-built bias benchmark for question answering,Protected class stereotypes,,BBQ: A hand-built bias benchmark for question answering,https://aclanthology.org/2022.findings-acl.165.pdf
-Text,"BBNLI, bias in NLI benchmark",Protected class stereotypes,,On Measuring Social Biases in Prompt-Based Multi-Task Learning,https://aclanthology.org/2022.findings-naacl.42.pdf
-Text,WinoGender,Bias between gender and occupation,,Gender Bias in Coreference Resolution,https://arxiv.org/abs/1804.09301
-Text,WinoQueer,"Bias between gender, sexuality",,Winoqueer,https://arxiv.org/abs/2306.15087
-Text,Level of caricature,,,CoMPosT: Characterizing and Evaluating Caricature in LLM Simulations,https://arxiv.org/abs/2310.11501
-Text,SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models,,,[2305.11840] SeeGULL: A Stereotype Benchmark with Broad Geo-Cultural Coverage Leveraging Generative Models,https://arxiv.org/abs/2305.11840
-Text,"Investigating Subtler Biases in LLMs:
-Ageism, Beauty, Institutional, and Nationality Bias in Generative Models",,,https://arxiv.org/abs/2309.08902,https://arxiv.org/abs/2309.08902
-Text,ROBBIE: Robust Bias Evaluation of Large Generative Language Models,,,[2311.18140] ROBBIE: Robust Bias Evaluation of Large Generative Language Models,https://arxiv.org/abs/2311.18140
-Image,Image Embedding Association Test (iEAT),Embedding associations,,"Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases | Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency",https://dl.acm.org/doi/abs/10.1145/3442188.3445932
-Image,Dataset leakage and model leakage,Gender and label bias,,[1811.08489] Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations,https://arxiv.org/abs/1811.08489
-Image,Grounded-WEAT,Joint vision and language embeddings,,Measuring Social Biases in Grounded Vision and Language Embeddings - ACL Anthology,https://aclanthology.org/2021.naacl-main.78/
-Image,Grounded-SEAT,,,,
-Image,CLIP-based evaluation,"Gender and race and class associations with four attribute categories (profession, political, object, and other.)",,DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers,https://arxiv.org/abs/2202.04053
-Image,Human evaluation,,,,
-Image,,,,[2108.02818] Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications,https://arxiv.org/abs/2108.02818
-Image,,,,[2004.07173] Bias in Multimodal AI: Testbed for Fair Automatic Recruitment,https://arxiv.org/abs/2004.07173
-Image,Characterizing the variation in generated images,,,Stable bias: Analyzing societal representations in diffusion models,https://arxiv.org/abs/2303.11408
-Image,Stereotypical representation of professions,,,Editing Implicit Assumptions in Text-to-Image Diffusion Models see section 6,
-Image,Effect of different scripts on text-to-image generation,"It evaluates generated images for cultural stereotypes, when using different scripts (homoglyphs). It somewhat measures the suceptibility of a model to produce cultural stereotypes by simply switching the script",,Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis,https://arxiv.org/pdf/2209.08891.pdf
-Image,Automatic classification of the immorality of images,,,Ensuring Visual Commonsense Morality for Text-to-Image Generation,https://arxiv.org/pdf/2209.08891.pdf
-Image,"ENTIGEN: effect on the
-diversity of the generated images when adding
-ethical intervention",,,"How well can Text-to-Image Generative Models understand Ethical
-Natural Language Interventions?",
-Image,Evaluating text-to-image models for (complex) biases,,,Easily accessible text-to-image generation amplifies demographic stereotypes at large scale,https://dl.acm.org/doi/abs/10.1145/3593013.3594095
-Image,,,,FACET: Fairness in Computer Vision Evaluation Benchmark,https://openaccess.thecvf.com/content/ICCV2023/html/Gustafson_FACET_Fairness_in_Computer_Vision_Evaluation_Benchmark_ICCV_2023_paper.html
-Image,Evaluating text-to-image models for occupation-gender biases from source to output,"Measuring bias from source to output (dataset, model and outcome). Using different prompts to search dataset and to generate images. Evaluate them in turn for stereotypes.","Evaluating for social attributes that one self-identifies for, e.g. gender, is challenging in computer- generated images.",Fair Diffusion: Instructing Text-to-Image Generation Models on Fairness,https://arxiv.org/abs/2302.10893
-Image,Evaluating text-to-image models for gender biases in a multilingual setting,Using different prompts in different languages to generate images and evaluate them in turn for stereotypes.,,Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You,
-Image,Evaluating text-to-image models for biases when multiple people are generated,This work focuses on generating images depicting multiple people. This puts the evaluation on a higher level beyond portrait evaluation.,Same as for the other evaluations of social attributes + evaluating for location in image is difficult as the models have no inherent spatial understanding.,The Male CEO and the Female Assistant: Probing Gender Biases in Text-To-Image Models Through Paired Stereotype Test,https://arxiv.org/abs/2402.11089
-Image,Multimodal Composite Association Score: Measuring Gender Bias in Generative Multimodal Models,Measure association between concepts in multi-modal settings (image and text),,,
-Image,VisoGender,"This work measures gender-occupation biases in image-to-text models by evaluating: (1) their ability to correctly resolve the pronouns of individuals in scenes, and (2) the perceived gender of individuals in images retrieved for gender-neutral search queries.",Relies on annotators’ perceptions of binary gender. Could better control for the fact that models generally struggle with captioning any scene that involves interactions between two or more individuals.,VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution,https://proceedings.neurips.cc/paper_files/paper/2023/hash/c93f26b1381b17693055a611a513f1e9-Abstract-Datasets_and_Benchmarks.html
-Audio,Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators,Lists harms of audio/speech generators,Not necessarily evaluation but a good source of taxonomy. We can use this to point readers towards high-level evaluations,https://arxiv.org/pdf/2402.01708.pdf,https://arxiv.org/pdf/2402.01708.pdf
-Video,Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on Networks,Human led evaluations of deepfakes to understand susceptibility and representational harms (including political violence),"Repr. harm, incite violence",https://arxiv.org/abs/2210.10026,https://arxiv.org/abs/2210.10026

DemoData.csv

@@ -0,0 +1,26 @@
+Group,Modality,Type,Metaname,Suggested Evaluation,What it is evaluating,Considerations,Link,URL,Screenshots,Applicable Models ,Datasets,Hashtags,Abstract,Authors
+BiasEvals,Text,Model,weat,Word Embedding Association Test (WEAT),Associations and word embeddings based on Implicit Associations Test (IAT),"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.",Semantics derived automatically from language corpora contain human-like biases,https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily,"['Images/WEAT1.png', 'Images/WEAT2.png']",,,"['Bias', 'Word Association', 'Embeddings', 'NLP']","Artificial intelligence and machine learning are in a period of astounding growth. However, there are concerns that these
+technologies may be used, either with or without intention, to perpetuate the prejudice and unfairness that unfortunately
+characterizes many human institutions. Here we show for the first time that human-like semantic biases result from the
+application of standard machine learning to ordinary language—the same sort of language humans are exposed to every
+day. We replicate a spectrum of standard human biases as exposed by the Implicit Association Test and other well-known
+psychological studies. We replicate these using a widely used, purely statistical machine-learning model—namely, the GloVe
+word embedding—trained on a corpus of text from the Web. Our results indicate that language itself contains recoverable and
+accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards
+race or gender, or even simply veridical, reflecting the status quo for the distribution of gender with respect to careers or first
+names. These regularities are captured by machine learning along with the rest of semantics. In addition to our empirical
+findings concerning language, we also contribute new methods for evaluating bias in text, the Word Embedding Association
+Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results have implications not only for AI and
+machine learning, but also for the fields of psychology, sociology, and human ethics, since they raise the possibility that mere
+exposure to everyday language can account for the biases we replicate here.","Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan"
+BiasEvals,Text,Dataset,stereoset,StereoSet,Protected class stereotypes,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,StereoSet: Measuring stereotypical bias in pretrained language models,https://arxiv.org/abs/2004.09456,,,,,,
+BiasEvals,Text,Dataset,crowspairs,Crow-S Pairs,Protected class stereotypes,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models,https://arxiv.org/abs/2010.00133,,,,,,
+BiasEvals,Text,Output,honest,HONEST: Measuring Hurtful Sentence Completion in Language Models,Protected class stereotypes and hurtful language,Automating stereotype detection makes distinguishing harmful stereotypes difficult. It also raises many false positives and can flag relatively neutral associations based in fact (e.g. population x has a high proportion of lactose intolerant people).,HONEST: Measuring Hurtful Sentence Completion in Language Models,https://aclanthology.org/2021.naacl-main.191.pdf,,,,,,
+BiasEvals,Image,Model,ieat,Image Embedding Association Test (iEAT),Embedding associations,"Although based in human associations, general societal attitudes do not always represent subgroups of people and cultures.","Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases | Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency",https://dl.acm.org/doi/abs/10.1145/3442188.3445932,,,,,,
+BiasEvals,Image,Dataset,imagedataleak,Dataset leakage and model leakage,Gender and label bias,,Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations,https://arxiv.org/abs/1811.08489,,,,,,
+BiasEvals,Image,Output,stablebias,Characterizing the variation in generated images,,,Stable bias: Analyzing societal representations in diffusion models,https://arxiv.org/abs/2303.11408,,,,,,
+BiasEvals,Image,Output,homoglyphbias,Effect of different scripts on text-to-image generation,"It evaluates generated images for cultural stereotypes, when using different scripts (homoglyphs). It somewhat measures the suceptibility of a model to produce cultural stereotypes by simply switching the script",,Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis,https://arxiv.org/pdf/2209.08891.pdf,,,,,,
+BiasEvals,Audio,Taxonomy,notmyvoice,Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators,Lists harms of audio/speech generators,Not necessarily evaluation but a good source of taxonomy. We can use this to point readers towards high-level evaluations,Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators,https://arxiv.org/pdf/2402.01708.pdf,,,,,,
+BiasEvals,Video,Output,videodiversemisinfo,Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on Networks,Human led evaluations of deepfakes to understand susceptibility and representational harms (including political violence),"Repr. harm, incite violence","Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on Networks
+",https://arxiv.org/abs/2210.10026,,,,,,
+Privacy,,,,,,,,,,,,,,

app.py

@@ -2,36 +2,80 @@ import gradio as gr
 from css import custom_css
 import pandas as pd
 from gradio_modal import Modal
+import os
+import yaml
+
+
+folder_path = 'configs'
+# List to store data from YAML files
+data_list = []
+metadata_dict = {}
+
+# Iterate over each file in the folder
+for filename in os.listdir(folder_path):
+    if filename.endswith('.yaml'):
+        # Read YAML file
+        file_path = os.path.join(folder_path, filename)
+        with open(file_path, 'r') as yamlfile:
+            yaml_data = yaml.safe_load(yamlfile)
+            # Append YAML data to list
+            data_list.append(yaml_data)
+            metadata_dict['<u>'+yaml_data['Link']+'</u>'] = yaml_data
+            
+globaldf = pd.DataFrame(data_list)
+globaldf['Link'] = '<u>'+globaldf['Link']+'</u>'
+
+# Define the desired order of categories
+modality_order = ["Text", "Image", "Audio", "Video"]
+type_order = ["Model", "Dataset", "Output", "Taxonomy"]
 
+# Convert Modality and Type columns to categorical with specified order
+globaldf['Modality'] = pd.Categorical(globaldf['Modality'], categories=modality_order, ordered=True)
+globaldf['Type'] = pd.Categorical(globaldf['Type'], categories=type_order, ordered=True)
 
+# Sort DataFrame by Modality and Type
+globaldf.sort_values(by=['Modality', 'Type'], inplace=True)
 
 # create a gradio page with tabs and accordions
 
 # Path: taxonomy.py
 
-metadatadict = {}
-
-def loadtable(path):
-    rawdf = pd.read_csv(path)
-    for i, row in rawdf.iterrows():
-        metadatadict['<u>'+str(row['Link'])+'</u>'] = '['+str(row['Link'])+']('+str(row['URL'])+')'
-    #rawdf['Link'] = '['+rawdf['Link']+']('+rawdf['URL']+')'
-    rawdf['Link'] = '<u>'+rawdf['Link']+'</u>'
-    rawdf = rawdf.drop(columns=['URL'])
-    return rawdf
+def filter_modality(filteredtable, modality_filter):
+    filteredtable = filteredtable[filteredtable['Modality'].isin(modality_filter)]
+    return filteredtable
 
-def filter_bias_table(fulltable, modality_filter):
-    filteredtable = fulltable[fulltable['Modality'].isin(modality_filter)]
+def filter_type(filteredtable, modality_filter):
+    filteredtable = filteredtable[filteredtable['Type'].isin(modality_filter)]
     return filteredtable
 
 def showmodal(evt: gr.SelectData):
     print(evt.value, evt.index, evt.target)
     modal = Modal(visible=False)
-    md = gr.Markdown("")
-    if evt.index[1] == 4:
+    titlemd = gr.Markdown("")
+    authormd = gr.Markdown("")
+    tagsmd = gr.Markdown("")
+    abstractmd = gr.Markdown("")
+    gallery = gr.Gallery([])
+    if evt.index[1] == 5:
         modal = Modal(visible=True)
-        md = gr.Markdown('# '+metadatadict[evt.value])
-    return [modal,md]
+        itemdic = metadata_dict[evt.value]
+
+        tags = itemdic['Hashtags']
+        if pd.notnull(tags) and len(tags)>0:
+            tagstr = ''
+            for tag in tags:
+                tagstr += '<span class="tag">#'+tag+'</span> '
+                tagsmd = gr.Markdown(tagstr, visible=True)
+
+        titlemd = gr.Markdown('# ['+itemdic['Link']+']('+itemdic['URL']+')',visible=True)
+
+        if pd.notnull(itemdic['Authors']):
+            authormd = gr.Markdown('## '+itemdic['Authors'],visible=True)
+        if pd.notnull(itemdic['Abstract']):
+            abstractmd = gr.Markdown(itemdic['Abstract'],visible=True)
+        if pd.notnull(itemdic['Screenshots']) and len(itemdic['Screenshots'])>0:
+            gallery = gr.Gallery(itemdic['Screenshots'],visible=True)
+    return [modal, titlemd, authormd, tagsmd, abstractmd, gallery]
 
 with gr.Blocks(title = "Social Impact Measurement V2", css=custom_css) as demo: #theme=gr.themes.Soft(), 
     # create tabs for the app, moving the current table to one titled "rewardbench" and the benchmark_text to a tab called "About"
@@ -51,7 +95,9 @@ The following categories are high-level, non-exhaustive, and present a synthesis
                     """)
     with gr.Tabs(elem_classes="tab-buttons") as tabs1:
         with gr.TabItem("Bias/Stereotypes"):
-            fulltable = loadtable('BiasEvals.csv')
+            fulltable = globaldf[globaldf['Group'] == 'BiasEvals']
+            fulltable = fulltable[['Modality','Type', 'Suggested Evaluation', 'What it is evaluating', 'Considerations', 'Link']]
+
             gr.Markdown("""
             Generative AI systems can perpetuate harmful biases from various sources, including systemic, human, and statistical biases. These biases, also known as "fairness" considerations, can manifest in the final system due to choices made throughout the development process. They include harmful associations and stereotypes related to protected classes, such as race, gender, and sexuality. Evaluating biases involves assessing correlations, co-occurrences, sentiment, and toxicity across different modalities, both within the model itself and in the outputs of downstream tasks.
                         """)
@@ -62,36 +108,25 @@ The following categories are high-level, non-exhaustive, and present a synthesis
                                                  show_label=True,
                                                 #  info="Which modality to show."
                                                  )
+                type_filter = gr.CheckboxGroup(["Model", "Dataset", "Output", "Taxonomy"], 
+                                                 value=["Model", "Dataset", "Output", "Taxonomy"], 
+                                                 label="Type", 
+                                                 show_label=True,
+                                                #  info="Which modality to show."
+                                                 )
             with gr.Row():
-                biastable_full = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=False)
-                biastable_filtered = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=True)
-                modality_filter.change(filter_bias_table, inputs=[biastable_full, modality_filter], outputs=biastable_filtered)
+                biastable_full = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=False, interactive=False)
+                biastable_filtered = gr.DataFrame(value=fulltable, wrap=True, datatype="markdown", visible=True, interactive=False)
+                modality_filter.change(filter_modality, inputs=[biastable_filtered, modality_filter], outputs=biastable_filtered)
+                type_filter.change(filter_type, inputs=[biastable_filtered, type_filter], outputs=biastable_filtered)
+
                 with Modal(visible=False) as modal:
-                    md = gr.Markdown("Test 1")
-                    gr.Markdown('### Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan')
-                    tags = ['Bias', 'Word Association', 'Embedding', 'NLP']
-                    tagmd = ''
-                    for tag in tags:
-                        tagmd += '<span class="tag">#'+tag+'</span> '
-                    gr.Markdown(tagmd)
-                    gr.Markdown('''
-Artificial intelligence and machine learning are in a period of astounding growth. However, there are concerns that these
-technologies may be used, either with or without intention, to perpetuate the prejudice and unfairness that unfortunately
-characterizes many human institutions. Here we show for the first time that human-like semantic biases result from the
-application of standard machine learning to ordinary language—the same sort of language humans are exposed to every
-day. We replicate a spectrum of standard human biases as exposed by the Implicit Association Test and other well-known
-psychological studies. We replicate these using a widely used, purely statistical machine-learning model—namely, the GloVe
-word embedding—trained on a corpus of text from the Web. Our results indicate that language itself contains recoverable and
-accurate imprints of our historic biases, whether these are morally neutral as towards insects or flowers, problematic as towards
-race or gender, or even simply veridical, reflecting the status quo for the distribution of gender with respect to careers or first
-names. These regularities are captured by machine learning along with the rest of semantics. In addition to our empirical
-findings concerning language, we also contribute new methods for evaluating bias in text, the Word Embedding Association
-Test (WEAT) and the Word Embedding Factual Association Test (WEFAT). Our results have implications not only for AI and
-machine learning, but also for the fields of psychology, sociology, and human ethics, since they raise the possibility that mere
-exposure to everyday language can account for the biases we replicate here.
-                                ''')
-                    gr.Gallery(['Images/WEAT1.png', 'Images/WEAT2.png'])
-                biastable_filtered.select(showmodal, None, [modal,md])
+                    titlemd = gr.Markdown(visible=False)
+                    authormd = gr.Markdown(visible=False)
+                    tagsmd = gr.Markdown(visible=False)
+                    abstractmd = gr.Markdown(visible=False)
+                    gallery = gr.Gallery(visible=False)
+                biastable_filtered.select(showmodal, None, [modal, titlemd, authormd, tagsmd, abstractmd, gallery])
 
 
 

configs/crowspairs.yaml

@@ -0,0 +1,17 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Automating stereotype detection makes distinguishing harmful stereotypes
+  difficult. It also raises many false positives and can flag relatively neutral associations
+  based in fact (e.g. population x has a high proportion of lactose intolerant people).
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language
+  Models'
+Modality: Text
+Screenshots: []
+Suggested Evaluation: Crow-S Pairs
+Type: Dataset
+URL: https://arxiv.org/abs/2010.00133
+What it is evaluating: Protected class stereotypes

configs/homoglyphbias.yaml

@@ -0,0 +1,16 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: .nan
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: Exploiting Cultural Biases via Homoglyphs in Text-to-Image Synthesis
+Modality: Image
+Screenshots: []
+Suggested Evaluation: Effect of different scripts on text-to-image generation
+Type: Output
+URL: https://arxiv.org/pdf/2209.08891.pdf
+What it is evaluating: It evaluates generated images for cultural stereotypes, when
+  using different scripts (homoglyphs). It somewhat measures the suceptibility of
+  a model to produce cultural stereotypes by simply switching the script

configs/honest.yaml

@@ -0,0 +1,16 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Automating stereotype detection makes distinguishing harmful stereotypes
+  difficult. It also raises many false positives and can flag relatively neutral associations
+  based in fact (e.g. population x has a high proportion of lactose intolerant people).
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'HONEST: Measuring Hurtful Sentence Completion in Language Models'
+Modality: Text
+Screenshots: []
+Suggested Evaluation: 'HONEST: Measuring Hurtful Sentence Completion in Language Models'
+Type: Output
+URL: https://aclanthology.org/2021.naacl-main.191.pdf
+What it is evaluating: Protected class stereotypes and hurtful language

configs/ieat.yaml

@@ -0,0 +1,17 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Although based in human associations, general societal attitudes do
+  not always represent subgroups of people and cultures.
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: Image Representations Learned With Unsupervised Pre-Training Contain Human-like
+  Biases | Proceedings of the 2021 ACM Conference on Fairness, Accountability, and
+  Transparency
+Modality: Image
+Screenshots: []
+Suggested Evaluation: Image Embedding Association Test (iEAT)
+Type: Model
+URL: https://dl.acm.org/doi/abs/10.1145/3442188.3445932
+What it is evaluating: Embedding associations

configs/imagedataleak.yaml

@@ -0,0 +1,15 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: .nan
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in
+  Deep Image Representations'
+Modality: Image
+Screenshots: []
+Suggested Evaluation: Dataset leakage and model leakage
+Type: Dataset
+URL: https://arxiv.org/abs/1811.08489
+What it is evaluating: Gender and label bias

configs/notmyvoice.yaml

@@ -0,0 +1,16 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Not necessarily evaluation but a good source of taxonomy. We can use
+  this to point readers towards high-level evaluations
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech Generators
+Modality: Audio
+Screenshots: []
+Suggested Evaluation: Not My Voice! A Taxonomy of Ethical and Safety Harms of Speech
+  Generators
+Type: Taxonomy
+URL: https://arxiv.org/pdf/2402.01708.pdf
+What it is evaluating: Lists harms of audio/speech generators

configs/stablebias.yaml

@@ -0,0 +1,14 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: .nan
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'Stable bias: Analyzing societal representations in diffusion models'
+Modality: Image
+Screenshots: []
+Suggested Evaluation: Characterizing the variation in generated images
+Type: Output
+URL: https://arxiv.org/abs/2303.11408
+What it is evaluating: .nan

configs/stereoset.yaml

@@ -0,0 +1,16 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Automating stereotype detection makes distinguishing harmful stereotypes
+  difficult. It also raises many false positives and can flag relatively neutral associations
+  based in fact (e.g. population x has a high proportion of lactose intolerant people).
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'StereoSet: Measuring stereotypical bias in pretrained language models'
+Modality: Text
+Screenshots: []
+Suggested Evaluation: StereoSet
+Type: Dataset
+URL: https://arxiv.org/abs/2004.09456
+What it is evaluating: Protected class stereotypes

configs/videodiversemisinfo.yaml

@@ -0,0 +1,19 @@
+Abstract: .nan
+'Applicable Models ': .nan
+Authors: .nan
+Considerations: Repr. harm, incite violence
+Datasets: .nan
+Group: BiasEvals
+Hashtags: .nan
+Link: 'Diverse Misinformation: Impacts of Human Biases on Detection of Deepfakes on
+  Networks
+
+  '
+Modality: Video
+Screenshots: []
+Suggested Evaluation: 'Diverse Misinformation: Impacts of Human Biases on Detection
+  of Deepfakes on Networks'
+Type: Output
+URL: https://arxiv.org/abs/2210.10026
+What it is evaluating: Human led evaluations of deepfakes to understand susceptibility
+  and representational harms (including political violence)

configs/weat.yaml

@@ -0,0 +1,42 @@
+Abstract: "Artificial intelligence and machine learning are in a period of astounding\
+  \ growth. However, there are concerns that these\ntechnologies may be used, either\
+  \ with or without intention, to perpetuate the prejudice and unfairness that unfortunately\n\
+  characterizes many human institutions. Here we show for the first time that human-like\
+  \ semantic biases result from the\napplication of standard machine learning to ordinary\
+  \ language\u2014the same sort of language humans are exposed to every\nday. We replicate\
+  \ a spectrum of standard human biases as exposed by the Implicit Association Test\
+  \ and other well-known\npsychological studies. We replicate these using a widely\
+  \ used, purely statistical machine-learning model\u2014namely, the GloVe\nword embedding\u2014\
+  trained on a corpus of text from the Web. Our results indicate that language itself\
+  \ contains recoverable and\naccurate imprints of our historic biases, whether these\
+  \ are morally neutral as towards insects or flowers, problematic as towards\nrace\
+  \ or gender, or even simply veridical, reflecting the status quo for the distribution\
+  \ of gender with respect to careers or first\nnames. These regularities are captured\
+  \ by machine learning along with the rest of semantics. In addition to our empirical\n\
+  findings concerning language, we also contribute new methods for evaluating bias\
+  \ in text, the Word Embedding Association\nTest (WEAT) and the Word Embedding Factual\
+  \ Association Test (WEFAT). Our results have implications not only for AI and\n\
+  machine learning, but also for the fields of psychology, sociology, and human ethics,\
+  \ since they raise the possibility that mere\nexposure to everyday language can\
+  \ account for the biases we replicate here."
+'Applicable Models ': .nan
+Authors: Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan
+Considerations: Although based in human associations, general societal attitudes do
+  not always represent subgroups of people and cultures.
+Datasets: .nan
+Group: BiasEvals
+Hashtags:
+- Bias
+- Word Association
+- Embeddings
+- NLP
+Link: Semantics derived automatically from language corpora contain human-like biases
+Modality: Text
+Screenshots:
+- Images/WEAT1.png
+- Images/WEAT2.png
+Suggested Evaluation: Word Embedding Association Test (WEAT)
+Type: Model
+URL: https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily
+What it is evaluating: Associations and word embeddings based on Implicit Associations
+  Test (IAT)

temp.ipynb

@@ -0,0 +1,568 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df = pd.read_csv('DemoData.csv')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import yaml\n",
+    "import os\n",
+    "import ast\n",
+    "\n",
+    "# Create a folder to store YAML files if it doesn't exist\n",
+    "if not os.path.exists('configs'):\n",
+    "    os.makedirs('configs')\n",
+    "\n",
+    "# Iterate over each row in the DataFrame\n",
+    "for index, row in df.iterrows():\n",
+    "    # Extract Metaname and use it as the filename for YAML\n",
+    "    filename = str(row['Metaname']) + '.yaml'\n",
+    "    # Convert 'Screenshots' column to a Python list\n",
+    "    screenshots_list = None\n",
+    "    try:\n",
+    "\n",
+    "        screenshots_list = ast.literal_eval(row['Screenshots'])\n",
+    "    except:\n",
+    "        screenshots_list = []\n",
+    "    # Remove the 'Metaname' and 'Screenshots' columns from the data to be converted to YAML\n",
+    "    row_data = row.drop(['Metaname', 'Screenshots'])\n",
+    "    # Convert the remaining data to a dictionary\n",
+    "    data_dict = row_data.to_dict()\n",
+    "    # Add the 'Screenshots' list to the dictionary\n",
+    "    data_dict['Screenshots'] = screenshots_list\n",
+    "    # Write the data as YAML to a new file\n",
+    "    with open(os.path.join('configs', filename), 'w') as yamlfile:\n",
+    "        yaml.dump(data_dict, yamlfile, default_flow_style=False)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<div>\n",
+       "<style scoped>\n",
+       "    .dataframe tbody tr th:only-of-type {\n",
+       "        vertical-align: middle;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe tbody tr th {\n",
+       "        vertical-align: top;\n",
+       "    }\n",
+       "\n",
+       "    .dataframe thead th {\n",
+       "        text-align: right;\n",
+       "    }\n",
+       "</style>\n",
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "  <thead>\n",
+       "    <tr style=\"text-align: right;\">\n",
+       "      <th></th>\n",
+       "      <th>Group</th>\n",
+       "      <th>Modality</th>\n",
+       "      <th>Level</th>\n",
+       "      <th>Metaname</th>\n",
+       "      <th>Suggested Evaluation</th>\n",
+       "      <th>What it is evaluating</th>\n",
+       "      <th>Considerations</th>\n",
+       "      <th>Link</th>\n",
+       "      <th>URL</th>\n",
+       "      <th>Screenshots</th>\n",
+       "      <th>Applicable Models</th>\n",
+       "      <th>Datasets</th>\n",
+       "      <th>Hashtags</th>\n",
+       "    </tr>\n",
+       "  </thead>\n",
+       "  <tbody>\n",
+       "    <tr>\n",
+       "      <th>0</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Text</td>\n",
+       "      <td>Model</td>\n",
+       "      <td>weat</td>\n",
+       "      <td>Word Embedding Association Test (WEAT)</td>\n",
+       "      <td>Associations and word embeddings based on Impl...</td>\n",
+       "      <td>Although based in human associations, general ...</td>\n",
+       "      <td>Semantics derived automatically from language ...</td>\n",
+       "      <td>https://researchportal.bath.ac.uk/en/publicati...</td>\n",
+       "      <td>['Images/WEAT1.png', 'Images/WEAT2.png']</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>1</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Text</td>\n",
+       "      <td>Model</td>\n",
+       "      <td>wefat</td>\n",
+       "      <td>Word Embedding Factual As\\nsociation Test (WEFAT)</td>\n",
+       "      <td>Associations and word embeddings based on Impl...</td>\n",
+       "      <td>Although based in human associations, general ...</td>\n",
+       "      <td>Semantics derived automatically from language ...</td>\n",
+       "      <td>https://researchportal.bath.ac.uk/en/publicati...</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>2</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Text</td>\n",
+       "      <td>Dataset</td>\n",
+       "      <td>stereoset</td>\n",
+       "      <td>StereoSet</td>\n",
+       "      <td>Protected class stereotypes</td>\n",
+       "      <td>Automating stereotype detection makes distingu...</td>\n",
+       "      <td>StereoSet: Measuring stereotypical bias in pre...</td>\n",
+       "      <td>https://arxiv.org/abs/2004.09456</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>3</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Text</td>\n",
+       "      <td>Dataset</td>\n",
+       "      <td>crwospairs</td>\n",
+       "      <td>Crow-S Pairs</td>\n",
+       "      <td>Protected class stereotypes</td>\n",
+       "      <td>Automating stereotype detection makes distingu...</td>\n",
+       "      <td>CrowS-Pairs: A Challenge Dataset for Measuring...</td>\n",
+       "      <td>https://arxiv.org/abs/2010.00133</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>4</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Text</td>\n",
+       "      <td>Output</td>\n",
+       "      <td>honest</td>\n",
+       "      <td>HONEST: Measuring Hurtful Sentence Completion ...</td>\n",
+       "      <td>Protected class stereotypes and hurtful language</td>\n",
+       "      <td>Automating stereotype detection makes distingu...</td>\n",
+       "      <td>HONEST: Measuring Hurtful Sentence Completion ...</td>\n",
+       "      <td>https://aclanthology.org/2021.naacl-main.191.pdf</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>5</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Image</td>\n",
+       "      <td>Model</td>\n",
+       "      <td>ieat</td>\n",
+       "      <td>Image Embedding Association Test (iEAT)</td>\n",
+       "      <td>Embedding associations</td>\n",
+       "      <td>Although based in human associations, general ...</td>\n",
+       "      <td>Image Representations Learned With Unsupervise...</td>\n",
+       "      <td>https://dl.acm.org/doi/abs/10.1145/3442188.344...</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>6</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Image</td>\n",
+       "      <td>Dataset</td>\n",
+       "      <td>imagedataleak</td>\n",
+       "      <td>Dataset leakage and model leakage</td>\n",
+       "      <td>Gender and label bias</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Balanced Datasets Are Not Enough: Estimating a...</td>\n",
+       "      <td>https://arxiv.org/abs/1811.08489</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>7</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Image</td>\n",
+       "      <td>Output</td>\n",
+       "      <td>stablebias</td>\n",
+       "      <td>Characterizing the variation in generated images</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Stable bias: Analyzing societal representation...</td>\n",
+       "      <td>https://arxiv.org/abs/2303.11408</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>8</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Image</td>\n",
+       "      <td>Output</td>\n",
+       "      <td>homoglyphbias</td>\n",
+       "      <td>Effect of different scripts on text-to-image g...</td>\n",
+       "      <td>It evaluates generated images for cultural ste...</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>Exploiting Cultural Biases via Homoglyphs in T...</td>\n",
+       "      <td>https://arxiv.org/pdf/2209.08891.pdf</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>9</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Audio</td>\n",
+       "      <td>Taxonomy (?)</td>\n",
+       "      <td>notmyvoice</td>\n",
+       "      <td>Not My Voice! A Taxonomy of Ethical and Safety...</td>\n",
+       "      <td>Lists harms of audio/speech generators</td>\n",
+       "      <td>Not necessarily evaluation but a good source o...</td>\n",
+       "      <td>Not My Voice! A Taxonomy of Ethical and Safety...</td>\n",
+       "      <td>https://arxiv.org/pdf/2402.01708.pdf</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>10</th>\n",
+       "      <td>BiasEvals</td>\n",
+       "      <td>Video</td>\n",
+       "      <td>Output</td>\n",
+       "      <td>videodiversemisinfo</td>\n",
+       "      <td>Diverse Misinformation: Impacts of Human Biase...</td>\n",
+       "      <td>Human led evaluations of deepfakes to understa...</td>\n",
+       "      <td>Repr. harm, incite violence</td>\n",
+       "      <td>Diverse Misinformation: Impacts of Human Biase...</td>\n",
+       "      <td>https://arxiv.org/abs/2210.10026</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "    <tr>\n",
+       "      <th>11</th>\n",
+       "      <td>Privacy</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "      <td>NaN</td>\n",
+       "    </tr>\n",
+       "  </tbody>\n",
+       "</table>\n",
+       "</div>"
+      ],
+      "text/plain": [
+       "        Group Modality         Level             Metaname  \\\n",
+       "0   BiasEvals     Text         Model                 weat   \n",
+       "1   BiasEvals     Text         Model                wefat   \n",
+       "2   BiasEvals     Text       Dataset            stereoset   \n",
+       "3   BiasEvals     Text       Dataset           crwospairs   \n",
+       "4   BiasEvals     Text        Output               honest   \n",
+       "5   BiasEvals    Image         Model                 ieat   \n",
+       "6   BiasEvals    Image       Dataset        imagedataleak   \n",
+       "7   BiasEvals    Image        Output           stablebias   \n",
+       "8   BiasEvals    Image        Output        homoglyphbias   \n",
+       "9   BiasEvals    Audio  Taxonomy (?)           notmyvoice   \n",
+       "10  BiasEvals    Video        Output  videodiversemisinfo   \n",
+       "11    Privacy      NaN           NaN                  NaN   \n",
+       "\n",
+       "                                 Suggested Evaluation  \\\n",
+       "0              Word Embedding Association Test (WEAT)   \n",
+       "1   Word Embedding Factual As\\nsociation Test (WEFAT)   \n",
+       "2                                           StereoSet   \n",
+       "3                                        Crow-S Pairs   \n",
+       "4   HONEST: Measuring Hurtful Sentence Completion ...   \n",
+       "5             Image Embedding Association Test (iEAT)   \n",
+       "6                   Dataset leakage and model leakage   \n",
+       "7    Characterizing the variation in generated images   \n",
+       "8   Effect of different scripts on text-to-image g...   \n",
+       "9   Not My Voice! A Taxonomy of Ethical and Safety...   \n",
+       "10  Diverse Misinformation: Impacts of Human Biase...   \n",
+       "11                                                NaN   \n",
+       "\n",
+       "                                What it is evaluating  \\\n",
+       "0   Associations and word embeddings based on Impl...   \n",
+       "1   Associations and word embeddings based on Impl...   \n",
+       "2                         Protected class stereotypes   \n",
+       "3                         Protected class stereotypes   \n",
+       "4    Protected class stereotypes and hurtful language   \n",
+       "5                              Embedding associations   \n",
+       "6                               Gender and label bias   \n",
+       "7                                                 NaN   \n",
+       "8   It evaluates generated images for cultural ste...   \n",
+       "9              Lists harms of audio/speech generators   \n",
+       "10  Human led evaluations of deepfakes to understa...   \n",
+       "11                                                NaN   \n",
+       "\n",
+       "                                       Considerations  \\\n",
+       "0   Although based in human associations, general ...   \n",
+       "1   Although based in human associations, general ...   \n",
+       "2   Automating stereotype detection makes distingu...   \n",
+       "3   Automating stereotype detection makes distingu...   \n",
+       "4   Automating stereotype detection makes distingu...   \n",
+       "5   Although based in human associations, general ...   \n",
+       "6                                                 NaN   \n",
+       "7                                                 NaN   \n",
+       "8                                                 NaN   \n",
+       "9   Not necessarily evaluation but a good source o...   \n",
+       "10                        Repr. harm, incite violence   \n",
+       "11                                                NaN   \n",
+       "\n",
+       "                                                 Link  \\\n",
+       "0   Semantics derived automatically from language ...   \n",
+       "1   Semantics derived automatically from language ...   \n",
+       "2   StereoSet: Measuring stereotypical bias in pre...   \n",
+       "3   CrowS-Pairs: A Challenge Dataset for Measuring...   \n",
+       "4   HONEST: Measuring Hurtful Sentence Completion ...   \n",
+       "5   Image Representations Learned With Unsupervise...   \n",
+       "6   Balanced Datasets Are Not Enough: Estimating a...   \n",
+       "7   Stable bias: Analyzing societal representation...   \n",
+       "8   Exploiting Cultural Biases via Homoglyphs in T...   \n",
+       "9   Not My Voice! A Taxonomy of Ethical and Safety...   \n",
+       "10  Diverse Misinformation: Impacts of Human Biase...   \n",
+       "11                                                NaN   \n",
+       "\n",
+       "                                                  URL  \\\n",
+       "0   https://researchportal.bath.ac.uk/en/publicati...   \n",
+       "1   https://researchportal.bath.ac.uk/en/publicati...   \n",
+       "2                    https://arxiv.org/abs/2004.09456   \n",
+       "3                    https://arxiv.org/abs/2010.00133   \n",
+       "4    https://aclanthology.org/2021.naacl-main.191.pdf   \n",
+       "5   https://dl.acm.org/doi/abs/10.1145/3442188.344...   \n",
+       "6                    https://arxiv.org/abs/1811.08489   \n",
+       "7                    https://arxiv.org/abs/2303.11408   \n",
+       "8                https://arxiv.org/pdf/2209.08891.pdf   \n",
+       "9                https://arxiv.org/pdf/2402.01708.pdf   \n",
+       "10                   https://arxiv.org/abs/2210.10026   \n",
+       "11                                                NaN   \n",
+       "\n",
+       "                                 Screenshots  Applicable Models   Datasets  \\\n",
+       "0   ['Images/WEAT1.png', 'Images/WEAT2.png']                 NaN       NaN   \n",
+       "1                                        NaN                 NaN       NaN   \n",
+       "2                                        NaN                 NaN       NaN   \n",
+       "3                                        NaN                 NaN       NaN   \n",
+       "4                                        NaN                 NaN       NaN   \n",
+       "5                                        NaN                 NaN       NaN   \n",
+       "6                                        NaN                 NaN       NaN   \n",
+       "7                                        NaN                 NaN       NaN   \n",
+       "8                                        NaN                 NaN       NaN   \n",
+       "9                                        NaN                 NaN       NaN   \n",
+       "10                                       NaN                 NaN       NaN   \n",
+       "11                                       NaN                 NaN       NaN   \n",
+       "\n",
+       "    Hashtags  \n",
+       "0        NaN  \n",
+       "1        NaN  \n",
+       "2        NaN  \n",
+       "3        NaN  \n",
+       "4        NaN  \n",
+       "5        NaN  \n",
+       "6        NaN  \n",
+       "7        NaN  \n",
+       "8        NaN  \n",
+       "9        NaN  \n",
+       "10       NaN  \n",
+       "11       NaN  "
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import urllib.request\n",
+    "from bs4 import BeautifulSoup\n",
+    "\n",
+    "from pypdf import PdfReader \n",
+    "from urllib.request import urlretrieve\n",
+    "\n",
+    "import pdfplumber\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily\n",
+      "\n",
+      "        Semantics derived automatically from language corpora contain human-like biases\n",
+      "     — the University of Bath's research portal\n",
+      "https://researchportal.bath.ac.uk/en/publications/semantics-derived-automatically-from-language-corpora-necessarily\n",
+      "\n",
+      "        Semantics derived automatically from language corpora contain human-like biases\n",
+      "     — the University of Bath's research portal\n",
+      "https://arxiv.org/abs/1903.10561\n",
+      "[1903.10561] On Measuring Social Biases in Sentence Encoders\n",
+      "https://dl.acm.org/doi/abs/10.5555/3454287.3455472\n",
+      "Error\n",
+      "https://arxiv.org/abs/2004.09456\n",
+      "[2004.09456] StereoSet: Measuring stereotypical bias in pretrained language models\n",
+      "https://arxiv.org/abs/2010.00133\n",
+      "[2010.00133] CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\n",
+      "https://aclanthology.org/2021.naacl-main.191.pdf\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "HONEST: Measuring Hurtful Sentence Completion in Language Models\n",
+      "nan\n",
+      "Error\n",
+      "https://aclanthology.org/2022.findings-acl.165.pdf\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "BBQ: A Hand-Built Bias Benchmark for Question Answering \n",
+      "https://aclanthology.org/2022.findings-naacl.42.pdf\n"
+     ]
+    },
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.\n"
+     ]
+    },
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "On Measuring Social Biases in Prompt-Based Multi-Task Learning\n"
+     ]
+    }
+   ],
+   "source": [
+    "def get_page_title(url):\n",
+    "    soup = BeautifulSoup(urllib.request.urlopen(url))\n",
+    "    return soup.title.string\n",
+    "\n",
+    "\n",
+    "def extract_pdf_title(url):\n",
+    "    urlretrieve(url, 'temp.pdf')\n",
+    "    with pdfplumber.open('temp.pdf') as pdf:\n",
+    "        for page in pdf.pages:\n",
+    "            for line in page.extract_text().split('\\n'):\n",
+    "                return line\n",
+    "        return \"\"\n",
+    "\n",
+    "    \n",
+    " \n",
+    "for url in df['URL'][:10]:\n",
+    "    try:\n",
+    "        print(url)\n",
+    "        title = get_page_title(url)\n",
+    "        print(title)\n",
+    "    except:\n",
+    "        try:\n",
+    "            title = extract_pdf_title(url)\n",
+    "            print(title)\n",
+    "        except:\n",
+    "            print(\"Error\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "gradio",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.12.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}