Tasks/Task 5/ft_chain_eval_results_df.csv

question,contexts,answer,ground_truth,faithfulness,answer_relevancy,context_recall,context_precision,answer_correctness
What procedures should be established and maintained for escalating GAI system incidents to the organizational risk management authority?,"['Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Priorities Related to Information Integrity Research and Development.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'times; and information about the procedures used to address requests for consideration along with the results \nof the evaluation of their accessibility. For systems used in sensitive domains, reporting should include infor\xad\nmation about training and governance procedures for these technologies. Reporting should also include docu\xad\nmentation of goals and assessment of meeting those goals, consideration of data included, and documentation \nof the governance of reasonable access to the technology. Reporting should be provided in a clear and \nmachine-readable manner. \n51']",I don't know.,Establish and maintain procedures for escalating GAI system incidents to the organizational risk management authority when specific criteria for deactivation or disengagement is met for a particular context of use or for the GAI system as a whole.,0.0,0.0,0.0,0.162499999991875,0.1806177835407171
How might sophisticated threat actors use GAI-powered security co-pilots in cybersecurity attacks?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'Information security for computer systems and data is a mature field with widely accepted and \nstandardized practices for offensive and defensive cyber capabilities. GAI-based systems present two \nprimary information security risks: GAI could potentially discover or enable new cybersecurity risks by \nlowering the barriers for or easing automated exercise of offensive capabilities; simultaneously, it \nexpands the available attack surface, as GAI itself is vulnerable to attacks like prompt injection or data \npoisoning.  \nOffensive cyber capabilities advanced by GAI systems may augment cybersecurity attacks such as \nhacking, malware, and phishing. Reports have indicated that LLMs are already able to discover some \nvulnerabilities in systems (hardware, software, data) and write code to exploit them. Sophisticated threat \nactors might further these risks by developing GAI-powered security co-pilots for use in several parts of'
 'harm, such as the ideation and design of novel harmful chemical or biological agents.  \nWhile some of these described capabilities lie beyond the reach of existing GAI tools, ongoing \nassessments of this risk would be enhanced by monitoring both the ability of AI tools to facilitate CBRN \nweapons planning and GAI systems’ connection or access to relevant data and tools. \nTrustworthy AI Characteristic: Safe, Explainable and Interpretable'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections']","Sophisticated threat actors might use GAI-powered security co-pilots to augment cybersecurity attacks such as hacking, malware, and phishing. These co-pilots could potentially discover vulnerabilities in systems (hardware, software, data) and write code to exploit them.",Sophisticated threat actors might use GAI-powered security co-pilots to inform attackers on how to proactively evade threat detection and escalate privileges after gaining system access.,1.0,1.0000000000000002,1.0,0.49999999995,0.4079478302316967
"What considerations should organizations take into account to ensure accessibility during the design, development, and deployment of automated systems?","['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'You should know that an automated system is being used, \nand understand how and why it contributes to outcomes \nthat impact you. Designers, developers, and deployers of automat\xad\ned systems should provide generally accessible plain language docu\xad\nmentation including clear descriptions of the overall system func\xad\ntioning and the role automation plays, notice that such systems are in \nuse, the individual or organization responsible for the system, and ex\xad\nplanations of outcomes that are clear, timely, and accessible. Such \nnotice should be kept up-to-date and people impacted by the system \nshould be notified of significant use case or key functionality chang\xad\nes. You should know how and why an outcome impacting you was de\xad\ntermined by an automated system, including when the automated \nsystem is not the sole input determining the outcome. Automated \nsystems should provide explanations that are technically valid, \nmeaningful and useful to you and to any operators or others who'
 'to proactively protect you from harms stemming from unintended, \nyet foreseeable, uses or impacts of automated systems. You should be \nprotected from inappropriate or irrelevant data use in the design, de\xad\nvelopment, and deployment of automated systems, and from the \ncompounded harm of its reuse. Independent evaluation and report\xad\ning that confirms that the system is safe and effective, including re\xad\nporting of steps taken to mitigate potential harms, should be per\xad\nformed and the results made public whenever possible. \n15'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'times; and information about the procedures used to address requests for consideration along with the results \nof the evaluation of their accessibility. For systems used in sensitive domains, reporting should include infor\xad\nmation about training and governance procedures for these technologies. Reporting should also include docu\xad\nmentation of goals and assessment of meeting those goals, consideration of data included, and documentation \nof the governance of reasonable access to the technology. Reporting should be provided in a clear and \nmachine-readable manner. \n51'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'or unintended uses lead to unintended harms. \nMany of the harms resulting from these technologies are preventable, and actions are already being taken to protect \nthe public. Some companies have put in place safeguards that have prevented harm from occurring by ensuring that \nkey development decisions are vetted by an ethics review; others have identified and mitigated harms found through \npre-deployment testing and ongoing monitoring processes. Governments at all levels have existing public consulta\xad\ntion processes that may be applied when considering the use of new automated systems, and existing product develop\xad\nment and testing practices already protect the American public from many potential harms. \nStill, these kinds of practices are deployed too rarely and unevenly. Expanded, proactive protections could build on \nthese existing practices, increase confidence in the use of automated systems, and protect the American public. Inno\xad']","Organizations should consider the following to ensure accessibility during the design, development, and deployment of automated systems:

1. **Accessibility and reasonable accommodations**: Ensuring that the systems are accessible to all users, including those with disabilities, and providing reasonable accommodations as needed.
2. **AI actor credentials and qualifications**: Ensuring that the individuals involved in the design, development, and deployment of the systems have the necessary credentials and qualifications.
3. **Alignment to organizational values**: Ensuring that the systems align with the organization's values.
4. **Auditing and assessment**: Implementing auditing and assessment processes to evaluate the systems.
5. **Change-management controls**: Establishing controls to manage changes in the systems.
6. **Commercial use**: Considering the implications of commercial use of the systems.
7. **Data provenance**: Ensuring the provenance of the data used in the systems.

These considerations help ensure that automated systems are designed, developed, and deployed in a manner that is accessible and inclusive.","Organizations should ensure accessibility to people with disabilities during the design, development, and deployment of automated systems. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers to the use or effectiveness of the automated system.",0.5294117647058824,0.988234100068221,0.5,0.48214285713080357,0.8175418744012988
What actions are suggested for examining and documenting the privacy risk of an AI system?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'Priorities Related to Information Integrity Research and Development.'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022']",I don't know.,"The suggested actions for examining and documenting the privacy risk of an AI system include: 1. Conducting AI red-teaming to assess issues such as outputting of training data samples, reverse engineering, model extraction, membership inference risks, revealing biometric, confidential, copyrighted, licensed, patented, personal, proprietary, sensitive, or trade-marked information, and tracking or revealing location information of users or members of training datasets. 2. Engaging directly with end-users and other stakeholders to understand their expectations and concerns regarding content provenance and using this feedback to guide the design of provenance data-tracking techniques. 3. Verifying deduplication of GAI training data samples, particularly regarding synthetic data.",0.0,0.0,0.0,0.499999999975,0.17977236772294186
How can robust watermarking techniques and corresponding detectors be useful in GAI systems used for content creation?,"['Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54']",I don't know.,Robust watermarking techniques and corresponding detectors can be useful in GAI systems used for content creation to identify the source of content.,0.0,0.0,0.0,0.0,0.18652238925722214
What is the importance of public consultation in the development of automated systems?,"['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'whether they are genuinely helpful in solving an identified problem. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of \nthese technologies, panelists individually described the importance of: receiving community input into the \ndesign and use of technologies, public reporting on crucial elements of these systems, better notice and consent \nprocedures that ensure privacy based on context and use case, ability to opt-out of using these systems and \nreceive a fallback to a human process, providing explanations of decisions and how these systems work, the \nneed for governance including training in using these systems, ensuring the technological use cases are \ngenuinely related to the goal task and are locally validated to work, and the need for institution and protection \nof third party audits to ensure systems continue to be accountable and valid. \n57'
 'or unintended uses lead to unintended harms. \nMany of the harms resulting from these technologies are preventable, and actions are already being taken to protect \nthe public. Some companies have put in place safeguards that have prevented harm from occurring by ensuring that \nkey development decisions are vetted by an ethics review; others have identified and mitigated harms found through \npre-deployment testing and ongoing monitoring processes. Governments at all levels have existing public consulta\xad\ntion processes that may be applied when considering the use of new automated systems, and existing product develop\xad\nment and testing practices already protect the American public from many potential harms. \nStill, these kinds of practices are deployed too rarely and unevenly. Expanded, proactive protections could build on \nthese existing practices, increase confidence in the use of automated systems, and protect the American public. Inno\xad'
 'You should know that an automated system is being used, \nand understand how and why it contributes to outcomes \nthat impact you. Designers, developers, and deployers of automat\xad\ned systems should provide generally accessible plain language docu\xad\nmentation including clear descriptions of the overall system func\xad\ntioning and the role automation plays, notice that such systems are in \nuse, the individual or organization responsible for the system, and ex\xad\nplanations of outcomes that are clear, timely, and accessible. Such \nnotice should be kept up-to-date and people impacted by the system \nshould be notified of significant use case or key functionality chang\xad\nes. You should know how and why an outcome impacting you was de\xad\ntermined by an automated system, including when the automated \nsystem is not the sole input determining the outcome. Automated \nsystems should provide explanations that are technically valid, \nmeaningful and useful to you and to any operators or others who'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'monitoring, and governance structures for automated systems, but also go further to address unmet needs for change and offer \nconcrete directions for how those changes can be made. \n• Expectations about reporting are intended for the entity developing or using the automated system. The resulting reports can \nbe provided to the public, regulators, auditors, industry standards groups, or others engaged in independent review, and should \nbe made public as much as possible consistent with law, regulation, and policy, and noting that intellectual property, law \nenforcement, or national security considerations may prevent public release. Where public reports are not possible, the \ninformation should be provided to oversight bodies and privacy, civil liberties, or other ethics officers charged with safeguard \ning individuals’ rights. These reporting expectations are important for transparency, so the American people can have'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'system is introduced or a large change implemented. This consultation should directly engage diverse impact\xad\ned communities to consider concerns and risks that may be unique to those communities, or disproportionate\xad\nly prevalent or severe for them. The extent of this engagement and the form of outreach to relevant stakehold\xad\ners may differ depending on the specific automated system and development phase, but should include \nsubject matter, sector-specific, and context-specific experts as well as experts on potential impacts such as \ncivil rights, civil liberties, and privacy experts. For private sector applications, consultations before product \nlaunch may need to be confidential. Government applications, particularly law enforcement applications or \napplications that raise national security considerations, may require confidential or limited engagement based \non system sensitivities and preexisting oversight laws and structures. Concerns raised in this consultation']","The importance of public consultation in the development of automated systems includes:

1. **Identifying and Mitigating Harms**: Public consultation helps in identifying potential harms and concerns that may be unique to or disproportionately affect certain communities. This allows for the mitigation of these harms before the system is deployed.

2. **Ensuring Diverse Input**: Engaging diverse impacted communities ensures that a wide range of perspectives and risks are considered, which may not be apparent to the developers alone.

3. **Building Confidence**: Expanded, proactive protections and public consultation can increase confidence in the use of automated systems by ensuring that they are vetted and validated by a broader audience.

4. **Transparency and Accountability**: Public consultation processes contribute to transparency and accountability, as they involve subject matter experts, sector-specific experts, and experts on potential impacts such as civil rights, civil liberties, and privacy.

5. **Adapting to Context**: The extent and form of public consultation may differ depending on the specific automated system and its development phase, ensuring that the consultation is relevant and context-specific.

6. **Confidentiality Considerations**: For private sector applications, consultations before product launch may need to be confidential. Government applications, particularly those related to law enforcement or national security, may also require confidential or limited engagement based on system sensitivities and preexisting oversight laws and structures.

Overall, public consultation is crucial for ensuring that automated systems are developed and deployed in a manner that is ethical, transparent, and considerate of the diverse needs and potential impacts on different communities.","Public consultation is important in the development of automated systems because it ensures that the public is involved in the design, implementation, deployment, acquisition, and maintenance phases. This consultation emphasizes early-stage engagement before a system is introduced or a large change is implemented. It directly engages diverse impacted communities to consider concerns and risks unique to those communities or disproportionately prevalent or severe for them. The consultation should include subject matter, sector-specific, and context-specific experts, as well as experts on potential impacts such as civil rights, civil liberties, and privacy experts. Concerns raised in this consultation should be documented, and the automated system developers should reconsider the system based on this feedback.",0.9473684210526315,0.9873676467824567,1.0,0.8644345237987182,0.8498287007987215
What mechanisms should be created to provide protections for whistleblowers who report organizational violations or risks to public safety?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'Priorities Related to Information Integrity Research and Development.'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Companies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'whether they are genuinely helpful in solving an identified problem. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of \nthese technologies, panelists individually described the importance of: receiving community input into the \ndesign and use of technologies, public reporting on crucial elements of these systems, better notice and consent \nprocedures that ensure privacy based on context and use case, ability to opt-out of using these systems and \nreceive a fallback to a human process, providing explanations of decisions and how these systems work, the \nneed for governance including training in using these systems, ensuring the technological use cases are \ngenuinely related to the goal task and are locally validated to work, and the need for institution and protection \nof third party audits to ensure systems continue to be accountable and valid. \n57'
 ""technology may or may not be part of an effective set of mechanisms to achieve safety. Various panelists raised \nconcerns about the validity of these systems, the tendency of adverse or irrelevant data to lead to a replication of \nunjust outcomes, and the confirmation bias and tendency of people to defer to potentially inaccurate automated \nsystems. Throughout, many of the panelists individually emphasized that the impact of these systems on \nindividuals and communities is potentially severe: the systems lack individualization and work against the \nbelief that people can change for the better, system use can lead to the loss of jobs and custody of children, and \nsurveillance can lead to chilling effects for communities and sends negative signals to community members \nabout how they're viewed. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of""
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance']",I don't know.,"Create mechanisms to provide protections for whistleblowers who report, based on reasonable belief, when the organization violates relevant laws or poses a specific and empirically well-substantiated negative risk to public safety (or has already caused harm).",0.0,0.0,0.0,0.0,0.18231348244221784
What approaches are suggested for mapping AI technology and legal risks of its components?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'ethics, or risk management. The Technical Companion builds on this prior work to provide practical next \nsteps to move these principles into practice and promote common approaches that allow technological \ninnovation to flourish while protecting people from harm. \n9'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,']",I don't know.,"Approaches for mapping AI technology and legal risks of its components include periodic monitoring of AI-generated content for privacy risks, implementing processes for responding to potential intellectual property infringement claims, connecting new GAI policies to existing governance and risk management activities, documenting training data curation policies, establishing policies for data collection and retention, implementing policies for the use and protection of third-party intellectual property and training data, re-evaluating models fine-tuned on third-party models, re-evaluating risks when adapting GAI models to new domains, and leveraging approaches to detect the presence of PII or sensitive data in generated output.",0.0,0.0,0.0,0.4105555555473445,0.17865687211082493
What is confabulation and how can it mislead or deceive users?,"['Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'it comes to open-ended prompts for long-form responses and in domains which require highly \ncontextual and/or domain expertise.  \nRisks from confabulations may arise when users believe false content – often due to the confident nature \nof the response – leading users to act upon or promote the false information. This poses a challenge for \nmany real-world applications, such as in healthcare, where a confabulated summary of patient \ninformation reports could cause doctors to make incorrect diagnoses and/or recommend the wrong \ntreatments. Risks of confabulated content may be especially important to monitor when integrating GAI \ninto applications involving consequential decision making. \nGAI outputs may also include confabulated logic or citations that purport to justify or explain the \nsystem’s answer, which may further mislead humans into inappropriately trusting the system’s output. \nFor instance, LLMs sometimes provide logical steps for how they arrived at an answer even when the'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'errors and other system flaws. These flaws were only revealed when an explanation of the system\nwas demanded and produced.86 The lack of an explanation made it harder for errors to be corrected in a\ntimely manner.\n42'
 'Priorities Related to Information Integrity Research and Development.'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022']","Confabulation refers to the generation of false content by AI systems, often presented with a high degree of confidence. This can mislead or deceive users because they may believe the false information and act upon it. For example, in healthcare, a confabulated summary of patient information could lead doctors to make incorrect diagnoses or recommend the wrong treatments. Additionally, AI systems may provide confabulated logic or citations that appear to justify or explain their answers, further misleading users into inappropriately trusting the system's output.",Confabulation is the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived.,1.0,0.9115600942510714,1.0,0.16666666665,0.9035312041570269
"What is the purpose of incorporating trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems according to the AI Risk Management Framework (AI RMF) 1.0?","['lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 '57 \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix B: \nHow AI Risks Differ from Traditional Software Risks. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B \nNational Institute of Standards and Technology (2023) AI RMF Playbook. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook \nNational Institue of Standards and Technology (2023) Framing Risk \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/1-sec-risk \nNational Institute of Standards and Technology (2023) The Language of Trustworthy AI: An In-Depth \nGlossary of Terms https://airc.nist.gov/AI_RMF_Knowledge_Base/Glossary \nNational Institue of Standards and Technology (2022) Towards a Standard for Identifying and Managing \nBias in Artificial Intelligence https://www.nist.gov/publications/towards-standard-identifying-and-\nmanaging-bias-artificial-intelligence'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'https://www.bloomberg.com/graphics/2023-generative-ai-bias/. \nNational Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \nNational Institute of Standards and Technology (2023) AI Risk Management Framework. \nhttps://www.nist.gov/itl/ai-risk-management-framework \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \nRisks and Trustworthiness. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \nDescriptions of AI Actor Tasks.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'Priorities Related to Information Integrity Research and Development.']",I don't know.,"The purpose of incorporating trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems according to the AI Risk Management Framework (AI RMF) 1.0 is to improve the ability of organizations to manage AI risks in a manner that is well-aligned with their goals, considers legal/regulatory requirements and best practices, and reflects risk management priorities.",0.0,0.0,0.0,0.4499999999775,0.17933560378232244
"What types of research does the National Science Foundation (NSF) fund to advance the safety, security, and effectiveness of AI systems?","['ENDNOTES\n23. National Science Foundation. National Artificial Intelligence Research Institutes. Accessed Sept. 12,\n2022. https://beta.nsf.gov/funding/opportunities/national-artificial-intelligence-research-institutes\n24. National Science Foundation. Cyber-Physical Systems. Accessed Sept. 12, 2022. https://beta.nsf.gov/\nfunding/opportunities/cyber-physical-systems-cps\n25. National Science Foundation. Secure and Trustworthy Cyberspace. Accessed Sept. 12, 2022. https://\nbeta.nsf.gov/funding/opportunities/secure-and-trustworthy-cyberspace-satc\n26. National Science Foundation. Formal Methods in the Field. Accessed Sept. 12, 2022. https://\nbeta.nsf.gov/funding/opportunities/formal-methods-field-fmitf\n27. National Science Foundation. Designing Accountable Software Systems. Accessed Sept. 12, 2022.\nhttps://beta.nsf.gov/funding/opportunities/designing-accountable-software-systems-dass\n28. The Leadership Conference Education Fund. The Use Of Pretrial “Risk Assessment” Instruments: A'
 'accuracy), and enable human users to understand, appropriately trust, and effectively manage the emerging \ngeneration of artificially intelligent partners.95 The National Science Foundation’s program on Fairness in \nArtificial Intelligence also includes a specific interest in research foundations for explainable AI.96\n45'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'Priorities Related to Information Integrity Research and Development.'
 'https://www.bloomberg.com/graphics/2023-generative-ai-bias/. \nNational Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \nNational Institute of Standards and Technology (2023) AI Risk Management Framework. \nhttps://www.nist.gov/itl/ai-risk-management-framework \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \nRisks and Trustworthiness. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \nDescriptions of AI Actor Tasks.'
 '57 \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix B: \nHow AI Risks Differ from Traditional Software Risks. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B \nNational Institute of Standards and Technology (2023) AI RMF Playbook. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook \nNational Institue of Standards and Technology (2023) Framing Risk \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/1-sec-risk \nNational Institute of Standards and Technology (2023) The Language of Trustworthy AI: An In-Depth \nGlossary of Terms https://airc.nist.gov/AI_RMF_Knowledge_Base/Glossary \nNational Institue of Standards and Technology (2022) Towards a Standard for Identifying and Managing \nBias in Artificial Intelligence https://www.nist.gov/publications/towards-standard-identifying-and-\nmanaging-bias-artificial-intelligence']","Based on the provided context, the National Science Foundation (NSF) funds research in the following areas to advance the safety, security, and effectiveness of AI systems:

1. National Artificial Intelligence Research Institutes
2. Cyber-Physical Systems
3. Secure and Trustworthy Cyberspace
4. Formal Methods in the Field
5. Designing Accountable Software Systems
6. Fairness in Artificial Intelligence, with a specific interest in research foundations for explainable AI

These areas encompass various aspects of AI, including fairness, accountability, security, and trustworthiness.","The National Science Foundation (NSF) funds extensive research to help foster the development of automated systems that adhere to and advance their safety, security, and effectiveness. Multiple NSF programs support research that directly addresses many of these principles: the National AI Research Institutes support research on all aspects of safe, trustworthy, fair, and explainable AI algorithms and systems; the Cyber Physical Systems program supports research on developing safe autonomous and cyber physical systems with AI components; the Secure and Trustworthy Cyberspace program supports research on cybersecurity and privacy enhancing technologies in automated systems; the Formal Methods in the Field program supports research on rigorous formal verification and analysis of automated systems and machine learning, and the Designing Accountable Software Systems program supports research on rigorous and reproducible methodologies for developing software systems with legal and regulatory compliance in mind.",1.0,0.9489662422924591,1.0,0.7182539682360118,0.4707464689755526
How have synthetic NCII and CSAM moved from niche internet forums to mainstream online businesses?,"['and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20'
 'imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it \nintended to imply that the entities, materials, or equipment are necessarily the best available for the \npurpose. Any mention of commercial, non-profit, academic partners, or their products, or references is \nfor information only; it is not intended to imply endorsement or recommendation by any U.S. \nGovernment agency.'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.']",I don't know.,"Websites, mobile apps, and custom-built models that generate synthetic NCII have moved from niche internet forums to mainstream, automated, and scaled online businesses.",0.0,0.0,0.0,0.0,0.1851748030212246
What measures are suggested to mitigate concerns of harmful bias and homogenization in AI training data?,"['organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'the information presented. \nHuman-AI Configuration; \nInformation Integrity \nMS-3.3-003 \nEvaluate potential biases and stereotypes that could emerge from the AI-\ngenerated content using appropriate methodologies including computational \ntesting methods as well as evaluating structured feedback input. \nHarmful Bias and Homogenization'
 '54 \nAppendix B. References \nAcemoglu, D. (2024) The Simple Macroeconomics of AI https://www.nber.org/papers/w32487 \nAI Incident Database. https://incidentdatabase.ai/ \nAtherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses. \nAI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/ \nBadyal, N. et al. (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611 \nBing Chat: Data Exfiltration Exploit Explained. Embrace The Red. \nhttps://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ \nBommasani, R. et al. (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome \nHomogenization? arXiv. https://arxiv.org/pdf/2211.13972 \nBoyarskaya, M. et al. (2020) Overcoming Failures of Imagination in AI Infused System Development and \nDeployment. arXiv. https://arxiv.org/pdf/2011.13416 \nBrowne, D. et al. (2023) Securing the AI Pipeline. Mandiant.']","The measures suggested to mitigate concerns of harmful bias and homogenization in AI training data include verifying deduplication of GAI (Generative Artificial Intelligence) training data samples, particularly regarding synthetic data, and evaluating potential biases and stereotypes that could emerge from the AI-generated content using appropriate methodologies, including computational testing methods as well as evaluating structured feedback input.",Assess the proportion of synthetic to non-synthetic training data and verify training data is not overly homogenous or GAI-produced to mitigate concerns of model collapse.,1.0,0.9999992167293253,1.0,0.7048611110934896,0.34751340119336405
How should organizational risk tolerances and controls be applied to third-party GAI resources?,"['Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Priorities Related to Information Integrity Research and Development.'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'ethics, or risk management. The Technical Companion builds on this prior work to provide practical next \nsteps to move these principles into practice and promote common approaches that allow technological \ninnovation to flourish while protecting people from harm. \n9'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']",I don't know.,"Organizational risk tolerances and controls should be applied to third-party GAI resources by incorporating them into acquisition and procurement processes, assessing personnel credentials and qualifications, performing background checks, filtering GAI input and outputs, grounding, fine-tuning, and retrieval-augmented generation. Additionally, organizational risk tolerance should be applied to the utilization of third-party datasets and other GAI resources, fine-tuned third-party models, and existing third-party models adapted to a new domain. Risk measurements should be reassessed after fine-tuning third-party GAI models.",0.0,0.0,0.0,0.8928571428348213,0.17859662721346473
What characteristics should data have to be considered representative and robust in the development or assessment of automated systems?,"['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'or label to ensure the goal of the automated system is appropriately identified and measured. Additionally, \njustification should be documented for each data attribute and source to explain why it is appropriate to use \nthat data to inform the results of the automated system and why such use will not violate any applicable laws. \nIn cases of high-dimensional and/or derived attributes, such justifications can be provided as overall \ndescriptions of the attribute generation process and appropriateness. \n19'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections']",I don't know.,Data used as part of system development or assessment should be representative of local communities based on the planned deployment setting and should be reviewed for bias based on the historical and societal context of the data. Such data should be sufficiently robust to identify and help to mitigate biases and potential harms.,0.0,0.0,1.0,0.3333333333,0.18190927667567255
What actions are suggested to ensure information integrity in the context of AI systems?,"['Priorities Related to Information Integrity Research and Development.'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles']",I don't know.,"The suggested actions to ensure information integrity in the context of AI systems include: 1) Employing methods to trace the origin and modifications of digital content. 2) Integrating tools designed to analyze content provenance and detect data anomalies, verify the authenticity of digital signatures, and identify patterns associated with misinformation or manipulation. 3) Disaggregating evaluation metrics by demographic factors to identify any discrepancies in how content provenance mechanisms work across diverse populations. 4) Evaluating novel methods and technologies for the measurement of GAI-related risks including in content provenance, offensive cyber, and CBRN, while maintaining the models’ ability to produce valid, reliable, and factually accurate outputs.",0.0,0.0,0.0,0.6984126983894179,0.18396607167756987
"What are the principles that federal agencies must adhere to under Executive Order 13960 when designing, developing, acquiring, or using AI?","['does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'in whole or in part may not be appropriate given the intended use of automated systems to achieve government \nagency missions. Future sector-specific guidance will likely be necessary and important for guiding the use of \nautomated systems in certain settings such as AI systems used as part of school building security or automated \nhealth diagnostic systems. \nThe Blueprint for an AI Bill of Rights recognizes that law enforcement activities require a balancing of \nequities, for example, between the protection of sensitive law enforcement information and the principle of \nnotice; as such, notice may not be appropriate, or may need to be adjusted to protect sources, methods, and \nother law enforcement equities. Even in contexts where these principles may not apply in whole or in part, \nfederal departments and agencies remain subject to judicial, privacy, and civil liberties oversight as well as \nexisting policies and safeguards that govern automated systems, including, for example, Executive Order 13960,'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'other countries. \nWe have seen modest progress in recent years, with some state and local governments responding to these prob\xad\nlems with legislation, and some courts extending longstanding statutory protections to new and emerging tech\xad\nnologies. There are companies working to incorporate additional protections in their design and use of auto\xad\nmated systems, and researchers developing innovative guardrails. Advocates, researchers, and government \norganizations have proposed principles for the ethical use of AI and other automated systems. These include \nthe Organization for Economic Co-operation and Development’s (OECD’s) 2019 Recommendation on Artificial \nIntelligence, which includes principles for responsible stewardship of trustworthy AI and which the United \nStates adopted, and Executive Order 13960 on Promoting the Use of Trustworthy Artificial Intelligence in the \nFederal Government, which sets out principles that govern the federal government’s use of AI. The Blueprint'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'Priorities Related to Information Integrity Research and Development.']",I don't know.,"The principles that federal agencies must adhere to under Executive Order 13960 when designing, developing, acquiring, or using AI are: (a) lawful and respectful of our Nation’s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d) safe, secure, and resilient; (e) understandable; (f) responsible and traceable; (g) regularly monitored; (h) transparent; and, (i) accountable.",0.0,0.0,1.0,0.38730158728867725,0.17863647133918925
How did the installation of a facial recognition system by a local public housing authority impact the community?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 ""technology may or may not be part of an effective set of mechanisms to achieve safety. Various panelists raised \nconcerns about the validity of these systems, the tendency of adverse or irrelevant data to lead to a replication of \nunjust outcomes, and the confirmation bias and tendency of people to defer to potentially inaccurate automated \nsystems. Throughout, many of the panelists individually emphasized that the impact of these systems on \nindividuals and communities is potentially severe: the systems lack individualization and work against the \nbelief that people can change for the better, system use can lead to the loss of jobs and custody of children, and \nsurveillance can lead to chilling effects for communities and sends negative signals to community members \nabout how they're viewed. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of""
 'Companies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32'
 'policing, automated license plate readers, real-time facial recognition systems (especially  \n    those used in public places or during protected activities like peaceful protests), social media  \n    monitoring, and ankle monitoring devices; \nVoting-related systems such as signature matching tools; \nSystems with a potential privacy impact such as smart home systems and associated data,  \n    systems that use or collect health-related data, systems that use or collect education-related  \n    data, criminal justice system data, ad-targeting systems, and systems that perform big data  \n    analytics in order to build profiles or infer personal information about individuals; and \nAny system that has the meaningful potential to lead to algorithmic discrimination. \n• Equal opportunities, including but not limited to:\nEducation-related systems such as algorithms that purport to detect student cheating or  \n    plagiarism, admissions algorithms, online or virtual reality student monitoring systems,'
 'evaluation and plain language reporting in the form of an algorithmic impact assessment, including \ndisparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'ethnicity, which can lead to race-based health inequities.47\n25\nAlgorithmic \nDiscrimination \nProtections']",I don't know.,"The installation of a facial recognition system by a local public housing authority led the community, both those living in the housing complex and not, to have videos of them sent to the local police department and made available for scanning by its facial recognition software.",0.0,0.0,0.0,0.49999999995,0.18457611981510857
How have businesses successfully integrated automated customer service with human support teams?,"['BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'health coverage.”107\nThe customer service industry has successfully integrated automated services such as \nchat-bots and AI-driven call response systems with escalation to a human support \nteam.108 Many businesses now use partially automated customer service platforms that help answer customer \nquestions and compile common problems for human agents to review. These integrated human-AI \nsystems allow companies to provide faster customer care while maintaining human agents to answer \ncalls or otherwise respond to complicated requests. Using both AI and human agents is viewed as key to \nsuccessful customer service.109\nBallot curing laws in at least 24 states require a fallback system that allows voters to \ncorrect their ballot and have it counted in the case that a voter signature matching \nalgorithm incorrectly flags their ballot as invalid or there is another issue with their \nballot, and review by an election official does not rectify the problem. Some federal'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']",Businesses have successfully integrated automated customer service with human support teams by using partially automated customer service platforms. These platforms help answer customer questions and compile common problems for human agents to review. This integration allows companies to provide faster customer care while maintaining human agents to answer calls or respond to complicated requests. Using both AI and human agents is viewed as key to successful customer service.,The customer service industry has successfully integrated automated services such as chat-bots and AI-driven call response systems with escalation to a human support team. Many businesses now use partially automated customer service platforms that help answer customer questions and compile common problems for human agents to review. These integrated human-AI systems allow companies to provide faster customer care while maintaining human agents to answer calls or otherwise respond to complicated requests. Using both AI and human agents is viewed as key to successful customer service.,1.0,0.9908724557828595,1.0,0.22499999998875,0.9948194187508337
Who were some of the private sector and civil society stakeholders that OSTP conducted meetings with for the development of the Blueprint for an AI Bill of Rights?,"['ing sessions, meetings, a formal request for information, and input to a publicly accessible and widely-publicized \nemail address, people throughout the United States, public servants across Federal agencies, and members of the \ninternational community spoke up about both the promises and potential harms of these technologies, and \nplayed a central role in shaping the Blueprint for an AI Bill of Rights. The core messages gleaned from these \ndiscussions include that AI has transformative potential to improve Americans’ lives, and that preventing the \nharms of these technologies is both necessary and achievable. The Appendix includes a full list of public engage-\nments. \n4'
 'APPENDIX\n• OSTP conducted meetings with a variety of stakeholders in the private sector and civil society. Some of these\nmeetings were specifically focused on providing ideas related to the development of the Blueprint for an AI\nBill of Rights while others provided useful general context on the positive use cases, potential harms, and/or\noversight possibilities for these technologies. Participants in these conversations from the private sector and\ncivil society included:\nAdobe \nAmerican Civil Liberties Union \n(ACLU) \nThe Aspen Commission on \nInformation Disorder \nThe Awood Center \nThe Australian Human Rights \nCommission \nBiometrics Institute \nThe Brookings Institute \nBSA | The Software Alliance \nCantellus Group \nCenter for American Progress \nCenter for Democracy and \nTechnology \nCenter on Privacy and Technology \nat Georgetown Law \nChristiana Care \nColor of Change \nCoworker \nData Robot \nData Trust Alliance \nData and Society Research Institute \nDeepmind \nEdSAFE AI Alliance \nElectronic Privacy Information'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']","Some of the private sector and civil society stakeholders that OSTP conducted meetings with for the development of the Blueprint for an AI Bill of Rights included:

- Adobe
- American Civil Liberties Union (ACLU)
- The Aspen Commission on Information Disorder
- The Awood Center
- The Australian Human Rights Commission
- Biometrics Institute
- The Brookings Institute
- BSA | The Software Alliance
- Cantellus Group
- Center for American Progress
- Center for Democracy and Technology
- Center on Privacy and Technology at Georgetown Law
- Christiana Care
- Color of Change
- Coworker
- Data Robot
- Data Trust Alliance
- Data and Society Research Institute
- Deepmind
- EdSAFE AI Alliance
- Electronic Privacy Information","Some of the private sector and civil society stakeholders that OSTP conducted meetings with for the development of the Blueprint for an AI Bill of Rights included Adobe, American Civil Liberties Union (ACLU), The Aspen Commission on Information Disorder, The Awood Center, The Australian Human Rights Commission, Biometrics Institute, The Brookings Institute, BSA | The Software Alliance, Cantellus Group, Center for American Progress, Center for Democracy and Technology, Center on Privacy and Technology at Georgetown Law, Christiana Care, Color of Change, Coworker, Data Robot, Data Trust Alliance, Data and Society Research Institute, Deepmind, EdSAFE AI Alliance, Electronic Privacy Information Center (EPIC), Encode Justice, Equal AI, Google, Hitachi's AI Policy Committee, The Innocence Project, Institute of Electrical and Electronics Engineers (IEEE), Intuit, Lawyers Committee for Civil Rights Under Law, Legal Aid Society, The Leadership Conference on Civil and Human Rights, Meta, Microsoft, The MIT AI Policy Forum, Movement Alliance Project, The National Association of Criminal Defense Lawyers, O’Neil Risk Consulting & Algorithmic Auditing, The Partnership on AI, Pinterest, The Plaintext Group, pymetrics, SAP, The Security Industry Association, Software and Information Industry Association (SIIA), Special Competitive Studies Project, Thorn, United for Respect, University of California at Berkeley Citris Policy Lab, University of California at Berkeley Labor Center, Unfinished/Project Liberty, Upturn, US Chamber of Commerce, US Chamber of Commerce Technology Engagement Center A.I. Working Group, Vibrent Health, Warehouse Worker Resource Center, and Waymap.",,0.973242051616895,1.0,0.49999999995,0.24589861625308068
What should be done if the residual negative risk of an AI system exceeds the organizational risk tolerance?,"['lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'Priorities Related to Information Integrity Research and Development.'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content']",I don't know.,Re-evaluate safety features of fine-tuned models when the negative risk exceeds organizational risk tolerance.,0.0,0.0,0.0,0.0,0.18244349869327503
What factors should be assessed to determine and document the expected and acceptable GAI system context of use?,"['Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it \nintended to imply that the entities, materials, or equipment are necessarily the best available for the \npurpose. Any mention of commercial, non-profit, academic partners, or their products, or references is \nfor information only; it is not intended to imply endorsement or recommendation by any U.S. \nGovernment agency.'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV']",I don't know.,"To determine and document the expected and acceptable GAI system context of use, the following factors should be assessed: Assumptions and limitations; Direct value to the organization; Intended operational environment and observed usage patterns; Potential positive and negative impacts to individuals, public safety, groups, communities, organizations, democratic institutions, and the physical environment; Social norms and expectations.",0.0,0.0,0.0,0.36666666664833336,0.1836481343264991
What is model collapse and what are its potential consequences?,"['and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'errors and other system flaws. These flaws were only revealed when an explanation of the system\nwas demanded and produced.86 The lack of an explanation made it harder for errors to be corrected in a\ntimely manner.\n42'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20']",I don't know.,"Model collapse can occur when model training over-relies on synthetic data, resulting in data points disappearing from the distribution of the new model's outputs. In addition to threatening the robustness of the model overall, model collapse could lead to homogenized outputs, including by amplifying any homogenization from the model used to generate the synthetic training data.",0.0,0.0,0.0,0.0,0.17883108386750768
What factors should be considered when decommissioning AI systems to ensure safety and maintain the organization's trustworthiness?,"['• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'Priorities Related to Information Integrity Research and Development.'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles']",I don't know.,"When decommissioning AI systems, the following factors should be considered to ensure safety and maintain the organization's trustworthiness: Data retention requirements; Data security, e.g., containment, protocols, Data leakage after decommissioning; Dependencies between upstream, downstream, or other data, internet of things (IOT) or AI systems; Use of open-source data or models; Users’ emotional entanglement with GAI functions.",0.0,0.0,0.0,0.36111111109907407,0.17929004840532906
What issues are associated with the tool meant to help low-risk federal prisoners win early release?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'Priorities Related to Information Integrity Research and Development.'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20'
 'Companies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,']",I don't know.,The tool meant to help low-risk federal prisoners win early release is plagued by flaws.,0.0,0.0,0.0,0.0,0.1851984875221609
How can GAI misuse cause emotional harm short and long-term?,"['Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'even if the inferences are not accurate (e.g., confabulations), and especially if they reveal information \nthat the individual considers sensitive or that is used to disadvantage or harm them. \nBeyond harms from information exposure (such as extortion or dignitary harm), wrong or inappropriate \ninferences of PII can contribute to downstream or secondary harmful impacts. For example, predictive \ninferences made by GAI models based on PII or protected attributes can contribute to adverse decisions, \nleading to representational or allocative harms to individuals or groups (see Harmful Bias and \nHomogenization below).'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'and a human who is interacting with it. Humans bring their unique perspectives, experiences, or domain-\nspecific expertise to interactions with AI systems but may not have detailed knowledge of AI systems and \nhow they work. As a result, human experts may be unnecessarily “averse” to GAI systems, and thus \ndeprive themselves or others of GAI’s beneficial uses.  \nConversely, due to the complexity and increasing reliability of GAI technology, over time, humans may \nover-rely on GAI systems or may unjustifiably perceive GAI content to be of higher quality than that \nproduced by other sources. This phenomenon is an example of automation bias, or excessive deference \nto automated systems. Automation bias can exacerbate other risks of GAI, such as risks of confabulation \nor risks of bias or homogenization. \nThere may also be concerns about emotional entanglement between humans and GAI systems, which \ncould lead to negative psychological impacts.']",I don't know.,"GAI misuse can cause emotional harm both immediately and over extended periods. Immediate emotional harm can result from the distribution of harmful deepfake images, while long-term emotional harm can stem from the effects of disinformation on societal trust in public institutions.",0.0,0.0,0.6666666666666666,0.3499999999825,0.18337395524353184
How is diverse feedback used in GAI evaluation to ensure provenance and reduce bias?,"['• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'evaluation and plain language reporting in the form of an algorithmic impact assessment, including \ndisparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36']","The context does not provide specific details on how diverse feedback is used in GAI evaluation to ensure provenance and reduce bias.

I don't know.","Diverse feedback is used in GAI evaluation to ensure provenance and reduce bias by recording and integrating structured feedback from operators, users, and potentially impacted communities through methods such as user research studies, focus groups, or community forums. This feedback is actively sought to assess content quality and potential biases, and the results are documented and incorporated into design, implementation, deployment approval, monitoring, and decommission decisions.",0.3333333333333333,0.0,0.0,0.0,0.3397933620174777
How did a store's misuse of predictive analytics show the need for better data protection?,"['external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Companies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32'
 'Priorities Related to Information Integrity Research and Development.'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022']",I don't know.,"A department store company used predictive analytics applied to collected consumer data to determine that a teenage girl was pregnant, and sent maternity clothing ads and other baby-related advertisements to her house, revealing to her father that she was pregnant.",0.0,0.0,0.0,0.249999999975,0.18508829933868393
How do safety metrics and security measures help AI handle and fix errors from threats?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 '54 \nAppendix B. References \nAcemoglu, D. (2024) The Simple Macroeconomics of AI https://www.nber.org/papers/w32487 \nAI Incident Database. https://incidentdatabase.ai/ \nAtherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses. \nAI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/ \nBadyal, N. et al. (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611 \nBing Chat: Data Exfiltration Exploit Explained. Embrace The Red. \nhttps://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ \nBommasani, R. et al. (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome \nHomogenization? arXiv. https://arxiv.org/pdf/2211.13972 \nBoyarskaya, M. et al. (2020) Overcoming Failures of Imagination in AI Infused System Development and \nDeployment. arXiv. https://arxiv.org/pdf/2011.13416 \nBrowne, D. et al. (2023) Securing the AI Pipeline. Mandiant.'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty']",I don't know.,"Safety metrics reflect system reliability and robustness, real-time monitoring, and response times for AI system failures. Security measures help assess vulnerabilities and threats, benchmark system security, gather user feedback, identify effective security metrics, measure content authentication methods, and perform AI red-teaming to assess resilience against various attacks.",0.0,0.0,1.0,0.22619047617916663,0.1780008643616345
How to ensure accessibility and equity in automated systems?,"['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 ""have the potential to meaningfully impact individuals' or communities' exercise of: \nRIGHTS, OPPORTUNITIES, OR ACCESS\nCivil rights, civil liberties, and privacy, including freedom of speech, voting, and protections from discrimi\xad\nnation, excessive punishment, unlawful surveillance, and violations of privacy and other freedoms in both \npublic and private sector contexts; \nEqual opportunities, including equitable access to education, housing, credit, employment, and other \nprograms; or, \nAccess to critical resources or services, such as healthcare, financial services, safety, social services, \nnon-deceptive information about goods and services, and government benefits. \nA list of examples of automated systems for which these principles should be considered is provided in the \nAppendix. The Technical Companion, which follows, offers supportive guidance for any person or entity that \ncreates, deploys, or oversees automated systems.""
 'You should know that an automated system is being used, \nand understand how and why it contributes to outcomes \nthat impact you. Designers, developers, and deployers of automat\xad\ned systems should provide generally accessible plain language docu\xad\nmentation including clear descriptions of the overall system func\xad\ntioning and the role automation plays, notice that such systems are in \nuse, the individual or organization responsible for the system, and ex\xad\nplanations of outcomes that are clear, timely, and accessible. Such \nnotice should be kept up-to-date and people impacted by the system \nshould be notified of significant use case or key functionality chang\xad\nes. You should know how and why an outcome impacting you was de\xad\ntermined by an automated system, including when the automated \nsystem is not the sole input determining the outcome. Automated \nsystems should provide explanations that are technically valid, \nmeaningful and useful to you and to any operators or others who'
 'The demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system.'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty']",I don't know.,"To ensure accessibility and equity in automated systems, organizations should design, develop, and deploy systems in ways that ensure accessibility to people with disabilities. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers. Additionally, automated systems should be tested using a broad set of measures to assess whether the system components produce disparities. Disparity assessments should include demographic performance measures, overall and subgroup parity assessment, and calibration. When disparities are identified, steps should be taken to mitigate or eliminate them, and ongoing monitoring should be conducted to assess algorithmic discrimination that might arise from unforeseen interactions or changes to the system or context of use.",0.0,0.0,1.0,0.5768253968138602,0.18070007468605156
How can orgs use public feedback and incident reports to improve GAI content in decision-making?,"['Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Priorities Related to Information Integrity Research and Development.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.']","Organizations can use public feedback and incident reports to improve GAI content in decision-making by implementing feedback activities that follow human subjects research requirements and best practices such as informed consent and subject compensation. Additionally, documentation practices including logging, recording, and analyzing GAI incidents can facilitate smoother sharing of information with relevant AI Actors. Regular information sharing, change management records, version history, and metadata can also empower AI Actors responding to and managing AI incidents.","Organizations can use public feedback and incident reports to improve GAI content in decision-making by integrating pre- and post-deployment external feedback into the monitoring process for GAI models and corresponding applications. This helps enhance awareness of performance changes and mitigate potential risks and harms from outputs. Additionally, documenting, reporting, and sharing information about GAI incidents can help mitigate and prevent harmful outcomes by assisting relevant AI Actors in tracing impacts to their source. Greater awareness and standardization of GAI incident reporting could promote transparency and improve GAI risk management across the AI ecosystem.",0.9333333333333333,0.9365147977562281,1.0,0.31944444443379627,0.36052977156806987
How do feedback and testing reveal GAI biases and societal impacts?,"['• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'evaluation and plain language reporting in the form of an algorithmic impact assessment, including \ndisparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.']",I don't know.,"Feedback and testing reveal GAI biases and societal impacts through methods such as user research studies, focus groups, community forums, adversarial testing, real-world scenario evaluations, and structured public feedback exercises. These methods help identify potential biases, understand misuse scenarios, and assess the general awareness among end users and impacted communities.",0.0,0.0,0.0,0.11111111109999999,0.18169426637419137
How do EO 13960 and NIST AI RMF ensure AI transparency and accountability for federal agencies?,"['lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 '57 \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix B: \nHow AI Risks Differ from Traditional Software Risks. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B \nNational Institute of Standards and Technology (2023) AI RMF Playbook. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook \nNational Institue of Standards and Technology (2023) Framing Risk \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/1-sec-risk \nNational Institute of Standards and Technology (2023) The Language of Trustworthy AI: An In-Depth \nGlossary of Terms https://airc.nist.gov/AI_RMF_Knowledge_Base/Glossary \nNational Institue of Standards and Technology (2022) Towards a Standard for Identifying and Managing \nBias in Artificial Intelligence https://www.nist.gov/publications/towards-standard-identifying-and-\nmanaging-bias-artificial-intelligence'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'https://www.bloomberg.com/graphics/2023-generative-ai-bias/. \nNational Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \nNational Institute of Standards and Technology (2023) AI Risk Management Framework. \nhttps://www.nist.gov/itl/ai-risk-management-framework \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \nRisks and Trustworthiness. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \nDescriptions of AI Actor Tasks.'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'Priorities Related to Information Integrity Research and Development.'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'Descriptions of AI Actor Tasks. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_A#:~:text=AI%20actors%\n20in%20this%20category,data%20providers%2C%20system%20funders%2C%20product']",I don't know.,"EO 13960 ensures AI transparency and accountability for federal agencies by requiring that AI is transparent and accountable among other principles. The NIST AI Risk Management Framework aims to foster the development of innovative approaches to address characteristics of trustworthiness, including transparency and accountability, during pre-design, design and development, deployment, use, and testing and evaluation of AI technologies and systems.",0.0,0.0,0.0,0.499999999975,0.17676853884076896
How can human expertise and content provenance boost GAI performance and ensure data privacy?,"['guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'and a human who is interacting with it. Humans bring their unique perspectives, experiences, or domain-\nspecific expertise to interactions with AI systems but may not have detailed knowledge of AI systems and \nhow they work. As a result, human experts may be unnecessarily “averse” to GAI systems, and thus \ndeprive themselves or others of GAI’s beneficial uses.  \nConversely, due to the complexity and increasing reliability of GAI technology, over time, humans may \nover-rely on GAI systems or may unjustifiably perceive GAI content to be of higher quality than that \nproduced by other sources. This phenomenon is an example of automation bias, or excessive deference \nto automated systems. Automation bias can exacerbate other risks of GAI, such as risks of confabulation \nor risks of bias or homogenization. \nThere may also be concerns about emotional entanglement between humans and GAI systems, which \ncould lead to negative psychological impacts.'
 'purpose specification. For example, most model developers do not disclose specific data sources on \nwhich models were trained, limiting user awareness of whether personally identifiably information (PII) \nwas trained on and, if so, how it was collected.  \nModels may leak, generate, or correctly infer sensitive information about individuals. For example, \nduring adversarial attacks, LLMs have revealed sensitive information (from the public domain) that was \nincluded in their training data. This problem has been referred to as data memorization, and may pose \nexacerbated privacy risks even for data present only in a small number of training samples.  \nIn addition to revealing sensitive information in GAI training data, GAI models may be able to correctly \ninfer PII or sensitive data that was not in their training data nor disclosed by the user by stitching \ntogether information from disparate sources. These inferences can have negative impact on an individual'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'MP-2.3-001 \nAssess the accuracy, quality, reliability, and authenticity of GAI output by \ncomparing it to a set of known ground truth data and by using a variety of \nevaluation methods (e.g., human oversight and automated evaluation, proven \ncryptographic techniques, review of content inputs). \nInformation Integrity']",I don't know.,"Human expertise and content provenance can boost GAI performance by employing techniques such as RLHF, fine-tuning, retrieval-augmented generation, content moderation, and business rules. To ensure data privacy, it is important to anonymize data, leverage privacy output filters, and remove any personally identifiable information (PII).",0.0,0.0,1.0,0.7961904761745524,0.18460049561184388
"What's the best environment for testing GAI's trustworthiness, data privacy, and human subject protection?","['external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Priorities Related to Information Integrity Research and Development.'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'MP-2.3-001 \nAssess the accuracy, quality, reliability, and authenticity of GAI output by \ncomparing it to a set of known ground truth data and by using a variety of \nevaluation methods (e.g., human oversight and automated evaluation, proven \ncryptographic techniques, review of content inputs). \nInformation Integrity'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36']",I don't know.,"The best environment for testing GAI's trustworthiness, data privacy, and human subject protection is a purpose-built testing environment such as NIST Dioptra.",0.0,0.0,0.0,0.0,0.18495814642243757
How can automated systems ensure accessibility and prevent bias?,"['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'The demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 ""technology may or may not be part of an effective set of mechanisms to achieve safety. Various panelists raised \nconcerns about the validity of these systems, the tendency of adverse or irrelevant data to lead to a replication of \nunjust outcomes, and the confirmation bias and tendency of people to defer to potentially inaccurate automated \nsystems. Throughout, many of the panelists individually emphasized that the impact of these systems on \nindividuals and communities is potentially severe: the systems lack individualization and work against the \nbelief that people can change for the better, system use can lead to the loss of jobs and custody of children, and \nsurveillance can lead to chilling effects for communities and sends negative signals to community members \nabout how they're viewed. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of""]",I don't know.,"Automated systems can ensure accessibility by being designed, developed, and deployed in ways that ensure accessibility to people with disabilities. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers. To prevent bias, automated systems should be tested using a broad set of measures to assess whether the system components produce disparities. This includes demographic performance measures, overall and subgroup parity assessment, and calibration. Disparity mitigation steps should be taken if a disparity is identified, and ongoing monitoring should be performed to assess algorithmic discrimination that might arise from unforeseen interactions or changes.",0.0,0.0,1.0,0.5638888888776111,0.1809630352708985
How do the U.S. AI Safety Institute and AI Risk Management Framework support the 2023 AI Executive Order?,"['BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 '57 \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix B: \nHow AI Risks Differ from Traditional Software Risks. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_B \nNational Institute of Standards and Technology (2023) AI RMF Playbook. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/Playbook \nNational Institue of Standards and Technology (2023) Framing Risk \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/1-sec-risk \nNational Institute of Standards and Technology (2023) The Language of Trustworthy AI: An In-Depth \nGlossary of Terms https://airc.nist.gov/AI_RMF_Knowledge_Base/Glossary \nNational Institue of Standards and Technology (2022) Towards a Standard for Identifying and Managing \nBias in Artificial Intelligence https://www.nist.gov/publications/towards-standard-identifying-and-\nmanaging-bias-artificial-intelligence'
 'https://www.bloomberg.com/graphics/2023-generative-ai-bias/. \nNational Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \nNational Institute of Standards and Technology (2023) AI Risk Management Framework. \nhttps://www.nist.gov/itl/ai-risk-management-framework \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \nRisks and Trustworthiness. \nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \nDescriptions of AI Actor Tasks.'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Priorities Related to Information Integrity Research and Development.'
 'About AI at NIST: The National Institute of Standards and Technology (NIST) develops measurements, \ntechnology, tools, and standards to advance reliable, safe, transparent, explainable, privacy-enhanced, \nand fair artificial intelligence (AI) so that its full commercial and societal benefits can be realized without \nharm to people or the planet. NIST, which has conducted both fundamental and applied work on AI for \nmore than a decade, is also helping to fulfill the 2023 Executive Order on Safe, Secure, and Trustworthy \nAI. NIST established the U.S. AI Safety Institute and the companion AI Safety Institute Consortium to \ncontinue the efforts set in motion by the E.O. to build the science necessary for safe, secure, and \ntrustworthy development and use of AI. \nAcknowledgments: This report was accomplished with the many helpful comments and contributions \nfrom the community, including the NIST Generative AI Public Working Group, and NIST staff and guest'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 '54 \nAppendix B. References \nAcemoglu, D. (2024) The Simple Macroeconomics of AI https://www.nber.org/papers/w32487 \nAI Incident Database. https://incidentdatabase.ai/ \nAtherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses. \nAI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/ \nBadyal, N. et al. (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611 \nBing Chat: Data Exfiltration Exploit Explained. Embrace The Red. \nhttps://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ \nBommasani, R. et al. (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome \nHomogenization? arXiv. https://arxiv.org/pdf/2211.13972 \nBoyarskaya, M. et al. (2020) Overcoming Failures of Imagination in AI Infused System Development and \nDeployment. arXiv. https://arxiv.org/pdf/2011.13416 \nBrowne, D. et al. (2023) Securing the AI Pipeline. Mandiant.']","The U.S. AI Safety Institute and the AI Risk Management Framework (AI RMF) support the 2023 AI Executive Order by advancing the science necessary for the safe, secure, and trustworthy development and use of AI. The National Institute of Standards and Technology (NIST) established the U.S. AI Safety Institute and the companion AI Safety Institute Consortium to continue efforts set in motion by the Executive Order, focusing on building reliable, safe, transparent, explainable, privacy-enhanced, and fair AI technologies.",The answer to given question is not present in context,1.0,0.9245110032267959,1.0,0.0,0.7738008646166127
How can we ensure effective oversight and risk mgmt of GAI systems with third-party input?,"['Table of Contents \n1. \nIntroduction ..............................................................................................................................................1 \n2. \nOverview of Risks Unique to or Exacerbated by GAI .....................................................................2 \n3. \nSuggested Actions to Manage GAI Risks ......................................................................................... 12 \nAppendix A. Primary GAI Considerations ............................................................................................... 47 \nAppendix B. References ................................................................................................................................ 54'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'whether they are genuinely helpful in solving an identified problem. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of \nthese technologies, panelists individually described the importance of: receiving community input into the \ndesign and use of technologies, public reporting on crucial elements of these systems, better notice and consent \nprocedures that ensure privacy based on context and use case, ability to opt-out of using these systems and \nreceive a fallback to a human process, providing explanations of decisions and how these systems work, the \nneed for governance including training in using these systems, ensuring the technological use cases are \ngenuinely related to the goal task and are locally validated to work, and the need for institution and protection \nof third party audits to ensure systems continue to be accountable and valid. \n57'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']",I don't know.,"To ensure effective oversight and risk management of GAI systems with third-party input, organizations should establish policies and procedures that address AI risks associated with third-party entities. This includes categorizing different types of GAI content with associated third-party rights, conducting joint educational activities and events to promote best practices for managing GAI risks, developing and validating approaches for measuring the success of content provenance management efforts, and drafting and maintaining well-defined contracts and service level agreements (SLAs) that specify content ownership, usage rights, quality standards, security requirements, and content provenance expectations.",0.0,0.0,0.4,0.3555555555437037,0.18059689442589597
How is the integrity of third-party pre-trained models ensured in GAI?,"['lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'listed for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.'
 'Priorities Related to Information Integrity Research and Development.']",I don't know.,"The integrity of third-party pre-trained models in GAI is ensured through several actions: reviewing transparency artifacts (e.g., system cards and model cards), applying explainable AI (XAI) techniques, documenting how pre-trained models have been adapted, documenting sources and types of training data and their origins, evaluating user-reported problematic content, implementing content filters, real-time monitoring processes, leveraging feedback from organizational boards or committees, using human moderation systems, and using organizational risk tolerance to evaluate acceptable risks and performance metrics.",0.0,0.0,0.0,0.2916666666520833,0.1823635468195095
How to ensure data accuracy and integrity in GAI systems while addressing impacts with AI Actors?,"['• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 'warrant additional human review, tracking and documentation, and greater management oversight.  \nAI technology can produce varied outputs in multiple modalities and present many classes of user \ninterfaces. This leads to a broader set of AI Actors interacting with GAI systems for widely differing \napplications and contexts of use. These can include data labeling and preparation, development of GAI \nmodels, content moderation, code generation and review, text generation and editing, image and video \ngeneration, summarization, search, and chat. These activities can take place within organizational \nsettings or in the public domain. \nOrganizations can restrict AI applications that cause harm, exceed stated risk tolerances, or that conflict \nwith their tolerances or values. Governance tools and protocols that are applied to other types of AI \nsystems can be applied to GAI systems. These plans and actions include: \n• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'Priorities Related to Information Integrity Research and Development.']",I don't know.,"To ensure data accuracy and integrity in GAI systems while addressing impacts with AI Actors, the following actions are suggested: 1) Plan regular engagements with AI Actors responsible for inputs to GAI systems, including third-party data and algorithms, to review and evaluate unanticipated impacts. 2) Review and document accuracy, representativeness, relevance, and suitability of data used at different stages of the AI life cycle. 3) Deploy and document fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems, especially when the information comes from multiple (or unknown) sources. 4) Develop and implement testing techniques to identify GAI-produced content that might be indistinguishable from human-generated content. 5) Implement plans for GAI systems to undergo regular adversarial testing to identify vulnerabilities and potential manipulation or misuse.",0.0,0.0,1.0,0.6726190476055952,0.18022231059059673
How do equity assessments and mitigation steps prevent algorithmic bias?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'evaluation and plain language reporting in the form of an algorithmic impact assessment, including \ndisparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5'
 'ethnicity, which can lead to race-based health inequities.47\n25\nAlgorithmic \nDiscrimination \nProtections'
 'an equitable way.  This protection should include proactive equity \nassessments as part of the system design, use of representative data \nand protection against proxies for demographic features, ensuring \naccessibility for people with disabilities in design and development, \npre-deployment and ongoing disparity testing and mitigation, and \nclear organizational oversight. Independent evaluation and plain \nlanguage reporting in the form of an algorithmic impact assessment, \nincluding disparity testing results and mitigation information, \nshould be performed and made public whenever possible to confirm \nthese protections.\n23'
 'protections should be built into their design, deployment, and ongoing use. \nMany companies, non-profits, and federal government agencies are already taking steps to ensure the public \nis protected from algorithmic discrimination. Some companies have instituted bias testing as part of their product \nquality assessment and launch procedures, and in some cases this testing has led products to be changed or not \nlaunched, preventing harm to the public. Federal government agencies have been developing standards and guidance \nfor the use of automated systems in order to help prevent bias. Non-profits and companies have developed best \npractices for audits and impact assessments to help identify potential algorithmic discrimination and provide \ntransparency to the public in the mitigation of such biases. \nBut there is much more work to do to protect the public from algorithmic discrimination to use and design'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 ""demographic information of impacted people is available, for example via testing with a sample of users or via \nqualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections""
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022']","Equity assessments and mitigation steps prevent algorithmic bias by incorporating several proactive measures into the system design and deployment process. These measures include:

1. **Proactive Equity Assessments**: Conducting equity assessments as part of the system design to identify and address potential biases early in the development process.
2. **Use of Representative Data**: Ensuring that the data used to train and operate the system is representative of the diverse populations it will impact, thereby reducing the risk of biased outcomes.
3. **Protection Against Proxies for Demographic Features**: Implementing safeguards to prevent the use of proxies that could indirectly introduce demographic biases.
4. **Accessibility for People with Disabilities**: Ensuring that the system is accessible to people with disabilities during both design and development phases.
5. **Pre-Deployment and Ongoing Disparity Testing and Mitigation**: Conducting disparity testing before deploying the system and continuing to test and mitigate disparities throughout its use.
6. **Clear Organizational Oversight**: Establishing clear oversight mechanisms within the organization to monitor and address any issues related to algorithmic bias.
7. **Independent Evaluation and Plain Language Reporting**: Performing independent evaluations and providing plain language reports, including disparity testing results and mitigation information, to confirm that protections against bias are effective and transparent.

These steps collectively help to identify, address, and mitigate potential biases, ensuring that automated systems operate in a fair and equitable manner.","Equity assessments and mitigation steps prevent algorithmic bias by conducting proactive equity assessments in the design phase to review potential input data, associated historical context, accessibility for people with disabilities, and societal goals to identify potential discrimination and effects on equity. Additionally, when a disparity assessment identifies a disparity against an assessed group, steps may be taken to mitigate or eliminate the disparity. This includes evaluating multiple models to select the one with the least adverse impact, modifying data input choices, or identifying a system with fewer disparities. If adequate mitigation is not possible, the use of the automated system should be reconsidered.",0.6086956521739131,0.9100656929237209,0.5,0.5311111111004889,0.8454491647022642
How do time-critical systems affect human fallback timing?,"['and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'errors and other system flaws. These flaws were only revealed when an explanation of the system\nwas demanded and produced.86 The lack of an explanation made it harder for errors to be corrected in a\ntimely manner.\n42'
 'and critical services. The American public deserves the assurance that, when rights, opportunities, or access are \nmeaningfully at stake and there is a reasonable expectation of an alternative to an automated system, they can conve\xad\nniently opt out of an automated system and will not be disadvantaged for that choice. In some cases, such a human or \nother alternative may be required by law, for example it could be required as “reasonable accommodations” for people \nwith disabilities. \nIn addition to being able to opt out and use a human alternative, the American public deserves a human fallback \nsystem in the event that an automated system fails or causes harm. No matter how rigorously an automated system is \ntested, there will always be situations for which the system fails. The American public deserves protection via human \nreview against these outlying or unexpected scenarios. In the case of time-critical systems, the public should not have'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']",I don't know.,"In time-critical systems, the human consideration and fallback mechanism should be immediately available or, where possible, available before the harm occurs. Time-critical systems include, but are not limited to, voting-related systems, automated building access and other access systems, systems that form a critical component of healthcare, and systems that have the ability to withhold wages or otherwise cause immediate financial penalties.",0.0,0.0,0.5,0.14285714284285714,0.18285583298127153
How do hiring algorithms and social media data impact democracy?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'ethnicity, which can lead to race-based health inequities.47\n25\nAlgorithmic \nDiscrimination \nProtections'
 'SECTION TITLE\xad\nFOREWORD\nAmong the great challenges posed to democracy today is the use of technology, data, and automated systems in \nways that threaten the rights of the American public. Too often, these tools are used to limit our opportunities and \nprevent our access to critical resources or services. These problems are well documented. In America and around \nthe world, systems supposed to help with patient care have proven unsafe, ineffective, or biased. Algorithms used \nin hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed \nnew harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s \nopportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or \nconsent. \nThese outcomes are deeply harmful—but they are not inevitable. Automated systems have brought about extraor-'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'Companies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'Priorities Related to Information Integrity Research and Development.'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content']","Based on the provided context, hiring algorithms and social media data impact democracy by limiting opportunities and preventing access to critical resources or services. Specifically, algorithms used in hiring decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination. Additionally, unchecked social media data collection has been used to threaten people’s opportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or consent. These outcomes are deeply harmful to democratic principles.

","Algorithms used in hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s opportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or consent.",1.0,0.9999999999999997,1.0,0.4166666666458333,0.9814428119959655
Who oversees assessment in automated systems?,"['detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'Priorities Related to Information Integrity Research and Development.'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'does not supersede, modify, or direct an interpretation of any existing statute, regulation, policy, or \ninternational instrument. It does not constitute binding guidance for the public or Federal agencies and \ntherefore does not require compliance with the principles described herein. It also is not determinative of what \nthe U.S. government’s position will be in any international negotiation. Adoption of these principles may not \nmeet the requirements of existing statutes, regulations, policies, or international instruments, or the \nrequirements of the Federal agencies that enforce them. These principles are not intended to, and do not, \nprohibit or limit any lawful activity of a government agency, including law enforcement, national security, or \nintelligence activities. \nThe appropriate application of the principles set forth in this white paper depends significantly on the \ncontext in which automated systems are being utilized. In some circumstances, application of these principles'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous']",I don't know.,Entities responsible for the development or use of automated systems should lay out clear governance structures and procedures. This includes responsibility of specific individuals or entities to oversee ongoing assessment and mitigation.,0.0,0.0,1.0,0.45833333331805554,0.181143320734443
How does diverse content in training materials impact societal AI?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 '56 \nKarasavva, V. et al. (2021) Personality, Attitudinal, and Demographic Predictors of Non-consensual \nDissemination of Intimate Images. NIH. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC9554400/ \nKatzman, J., et al. (2023) Taxonomizing and measuring representational harms: a look at image tagging. \nAAAI. https://dl.acm.org/doi/10.1609/aaai.v37i12.26670 \nKhan, T. et al. (2024) From Code to Consumer: PAI’s Value Chain Analysis Illuminates Generative AI’s Key \nPlayers. AI. https://partnershiponai.org/from-code-to-consumer-pais-value-chain-analysis-illuminates-\ngenerative-ais-key-players/ \nKirchenbauer, J. et al. (2023) A Watermark for Large Language Models. OpenReview. \nhttps://openreview.net/forum?id=aX8ig9X2a7 \nKleinberg, J. et al. (May 2021) Algorithmic monoculture and social welfare. PNAS. \nhttps://www.pnas.org/doi/10.1073/pnas.2018340118 \nLakatos, S. (2023) A Revealing Picture. Graphika. https://graphika.com/reports/a-revealing-picture']",I don't know.,The answer to given question is not present in context,0.0,0.0,1.0,0.0,0.19520361397610936
"Which methods use expert feedback, group input, or anonymous surveys?","['external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'decision making, and enhancing system documentation and debugging practices. When implementing \nfeedback activities, organizations should follow human subjects research requirements and best \npractices such as informed consent and subject compensation.'
 'results, and actions taken; and the procedures for and results from independent evaluations. Reporting \nshould be provided in a plain language and machine-readable manner. \n20'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'whether they are genuinely helpful in solving an identified problem. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of \nthese technologies, panelists individually described the importance of: receiving community input into the \ndesign and use of technologies, public reporting on crucial elements of these systems, better notice and consent \nprocedures that ensure privacy based on context and use case, ability to opt-out of using these systems and \nreceive a fallback to a human process, providing explanations of decisions and how these systems work, the \nneed for governance including training in using these systems, ensuring the technological use cases are \ngenuinely related to the goal task and are locally validated to work, and the need for institution and protection \nof third party audits to ensure systems continue to be accountable and valid. \n57'
 'detailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3']",I don't know.,"Participatory engagement methods use expert feedback, group input, or anonymous surveys.",0.0,0.0,0.0,0.11111111109999999,0.18572621750226204
Which metrics show AI reliability and failure response?,"['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'Priorities Related to Information Integrity Research and Development.'
 '54 \nAppendix B. References \nAcemoglu, D. (2024) The Simple Macroeconomics of AI https://www.nber.org/papers/w32487 \nAI Incident Database. https://incidentdatabase.ai/ \nAtherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses. \nAI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/ \nBadyal, N. et al. (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611 \nBing Chat: Data Exfiltration Exploit Explained. Embrace The Red. \nhttps://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ \nBommasani, R. et al. (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome \nHomogenization? arXiv. https://arxiv.org/pdf/2211.13972 \nBoyarskaya, M. et al. (2020) Overcoming Failures of Imagination in AI Infused System Development and \nDeployment. arXiv. https://arxiv.org/pdf/2011.13416 \nBrowne, D. et al. (2023) Securing the AI Pipeline. Mandiant.'
 'inputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'guide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'
 'Lakatos, S. (2023) A Revealing Picture. Graphika. https://graphika.com/reports/a-revealing-picture \nLee, H. et al. (2024) Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks. \narXiv. https://arxiv.org/pdf/2310.07879 \nLenaerts-Bergmans, B. (2024) Data Poisoning: The Exploitation of Generative AI. Crowdstrike. \nhttps://www.crowdstrike.com/cybersecurity-101/cyberattacks/data-poisoning/ \nLiang, W. et al. (2023) GPT detectors are biased against non-native English writers. arXiv. \nhttps://arxiv.org/abs/2304.02819 \nLuccioni, A. et al. (2023) Power Hungry Processing: Watts Driving the Cost of AI Deployment? arXiv. \nhttps://arxiv.org/pdf/2311.16863 \nMouton, C. et al. (2024) The Operational Risks of AI in Large-Scale Biological Attacks. RAND. \nhttps://www.rand.org/pubs/research_reports/RRA2977-2.html. \nNicoletti, L. et al. (2023) Humans Are Biased. Generative Ai Is Even Worse. Bloomberg. \nhttps://www.bloomberg.com/graphics/2023-generative-ai-bias/.']",I don't know.,"Safety metrics reflect system reliability and robustness, real-time monitoring, and response times for AI system failures.",0.0,0.0,0.0,0.249999999975,0.18170358254246638
How does watch list opacity impact error correction and public understanding?,"['for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content'
 'and data that are considered sensitive are understood to change over time based on societal norms and context. \n36'
 'evaluation and plain language reporting in the form of an algorithmic impact assessment, including \ndisparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5'
 'Priorities Related to Information Integrity Research and Development.'
 'Technology Policy Institute produced a synopsis of both the RFI submissions and the feedback at the listening\nsessions.115\n61'
 'and biological design tools. arXiv. https://arxiv.org/pdf/2306.13952'
 'FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12']",I don't know.,Watch list opacity impacts error correction and public understanding by making it difficult for both police and the public to understand why and how the system is making determinations. This lack of transparency can prevent errors from being corrected in a timely manner.,0.0,0.0,0.0,0.0,0.18446736094669172
"How do policies manage third-party AI risks, IP, and data privacy?","['FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12'
 'external use, narrow vs. broad application scope, fine-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation). \nData Privacy; Intellectual \nProperty'
 'BLUEPRINT FOR AN \nAI BILL OF \nRIGHTS \nMAKING AUTOMATED \nSYSTEMS WORK FOR \nTHE AMERICAN PEOPLE \nOCTOBER 2022'
 '• Accessibility and reasonable \naccommodations \n• AI actor credentials and qualifications  \n• Alignment to organizational values \n• Auditing and assessment \n• Change-management controls \n• Commercial use \n• Data provenance'
 'lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'
 'for any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections'
 'and management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.'
 'Priorities Related to Information Integrity Research and Development.'
 'organization’s business processes or other activities, system goals, any human-run procedures that form a \npart of the system, and specific performance expectations; a description of any data used to train machine \nlearning models or for other purposes, including how data sources were processed and interpreted, a \nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \nidentification and management assessments and any steps taken to mitigate potential harms; the results of \nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'
 'Information Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content']",I don't know.,"Policies manage third-party AI risks, IP, and data privacy by categorizing different types of GAI content with associated third-party rights (e.g., copyright, intellectual property, data privacy), conducting joint educational activities and events in collaboration with third parties to promote best practices for managing GAI risks, developing and validating approaches for measuring the success of content provenance management efforts with third parties (e.g., incidents detected and response times), and drafting and maintaining well-defined contracts and service level agreements (SLAs) that specify content ownership, usage rights, quality standards, security requirements, and content provenance expectations for GAI systems.",0.0,0.0,0.25,0.18253968253055555,0.18076821623726175