Tasks/Task 5/task5-ai-safety-sdg2.csv

question,contexts,ground_truth,evolution_type,metadata,episode_done
What procedures should be established and maintained for escalating GAI system incidents to the organizational risk management authority?,"[' \n42 \nMG-2.4-002 \nEstablish and maintain procedures for escalating GAI system incidents to the \norganizational risk management authority when specific criteria for deactivation \nor disengagement is met for a particular context of use or for the GAI system as a \nwhole. \nInformation Security \nMG-2.4-003 \nEstablish and maintain procedures for the remediation of issues which trigger \nincident response processes for the use of a GAI system, and provide stakeholders \ntimelines associated with the remediation plan. \nInformation Security \n \nMG-2.4-004 Establish and regularly review specific criteria that warrants the deactivation of \nGAI systems in accordance with set risk tolerances and appetites. \nInformation Security \n \nAI Actor Tasks: AI Deployment, Governance and Oversight, Operation and Monitoring \n \nMANAGE 3.1: AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied and \ndocumented. \nAction ID \nSuggested Action \nGAI Risks \nMG-3.1-001 \nApply organizational risk tolerances and controls (e.g., acquisition and \nprocurement processes; assessing personnel credentials and qualifications, \nperforming background checks; filtering GAI input and outputs, grounding, fine \ntuning, retrieval-augmented generation) to third-party GAI resources: Apply \norganizational risk tolerance to the utilization of third-party datasets and other \nGAI resources; Apply organizational risk tolerances to fine-tuned third-party \nmodels; Apply organizational risk tolerance to existing third-party models \nadapted to a new domain; Reassess risk measurements after fine-tuning third-\nparty GAI models. \nValue Chain and Component \nIntegration; Intellectual Property \nMG-3.1-002 \nTest GAI system value chain risks (e.g., data poisoning, malware, other software \nand hardware vulnerabilities; labor practices; data privacy and localization \ncompliance; geopolitical alignment). \nData Privacy; Information Security; \nValue Chain and Component \nIntegration; Harmful Bias and \nHomogenization \nMG-3.1-003 \nRe-assess model risks after fine-tuning or retrieval-augmented generation \nimplementation and for any third-party GAI models deployed for applications \nand/or use cases that were not evaluated in initial testing. \nValue Chain and Component \nIntegration \nMG-3.1-004 \nTake reasonable measures to review training data for CBRN information, and \nintellectual property, and where appropriate, remove it. Implement reasonable \nmeasures to prevent, flag, or take other action in response to outputs that \nreproduce particular training data (e.g., plagiarized, trademarked, patented, \nlicensed content or trade secret material). \nIntellectual Property; CBRN \nInformation or Capabilities \n']",Establish and maintain procedures for escalating GAI system incidents to the organizational risk management authority when specific criteria for deactivation or disengagement is met for a particular context of use or for the GAI system as a whole.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 45, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How might sophisticated threat actors use GAI-powered security co-pilots in cybersecurity attacks?,"[' \n10 \nGAI systems can ease the unintentional production or dissemination of false, inaccurate, or misleading \ncontent (misinformation) at scale, particularly if the content stems from confabulations.  \nGAI systems can also ease the deliberate production or dissemination of false or misleading information \n(disinformation) at scale, where an actor has the explicit intent to deceive or cause harm to others. Even \nvery subtle changes to text or images can manipulate human and machine perception. \nSimilarly, GAI systems could enable a higher degree of sophistication for malicious actors to produce \ndisinformation that is targeted towards specific demographics. Current and emerging multimodal models \nmake it possible to generate both text-based disinformation and highly realistic “deepfakes” – that is, \nsynthetic audiovisual content and photorealistic images.12 Additional disinformation threats could be \nenabled by future GAI models trained on new data modalities. \nDisinformation and misinformation – both of which may be facilitated by GAI – may erode public trust in \ntrue or valid evidence and information, with downstream effects. For example, a synthetic image of a \nPentagon blast went viral and briefly caused a drop in the stock market. Generative AI models can also \nassist malicious actors in creating compelling imagery and propaganda to support disinformation \ncampaigns, which may not be photorealistic, but could enable these campaigns to gain more reach and \nengagement on social media platforms. Additionally, generative AI models can assist malicious actors in \ncreating fraudulent content intended to impersonate others. \nTrustworthy AI Characteristics: Accountable and Transparent, Safe, Valid and Reliable, Interpretable and \nExplainable \n2.9. Information Security \nInformation security for computer systems and data is a mature field with widely accepted and \nstandardized practices for offensive and defensive cyber capabilities. GAI-based systems present two \nprimary information security risks: GAI could potentially discover or enable new cybersecurity risks by \nlowering the barriers for or easing automated exercise of offensive capabilities; simultaneously, it \nexpands the available attack surface, as GAI itself is vulnerable to attacks like prompt injection or data \npoisoning.  \nOffensive cyber capabilities advanced by GAI systems may augment cybersecurity attacks such as \nhacking, malware, and phishing. Reports have indicated that LLMs are already able to discover some \nvulnerabilities in systems (hardware, software, data) and write code to exploit them. Sophisticated threat \nactors might further these risks by developing GAI-powered security co-pilots for use in several parts of \nthe attack chain, including informing attackers on how to proactively evade threat detection and escalate \nprivileges after gaining system access. \nInformation security for GAI models and systems also includes maintaining availability of the GAI system \nand the integrity and (when applicable) the confidentiality of the GAI code, training data, and model \nweights. To identify and secure potential attack points in AI systems or specific components of the AI \n \n \n12 See also https://doi.org/10.6028/NIST.AI.100-4, to be published. \n']",Sophisticated threat actors might use GAI-powered security co-pilots to inform attackers on how to proactively evade threat detection and escalate privileges after gaining system access.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 13, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"What considerations should organizations take into account to ensure accessibility during the design, development, and deployment of automated systems?","["" \n \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEnsuring accessibility during design, development, and deployment. Systems should be \ndesigned, developed, and deployed by organizations in ways that ensure accessibility to people with disabili\xad\nties. This should include consideration of a wide variety of disabilities, adherence to relevant accessibility \nstandards, and user experience research both before and after deployment to identify and address any accessi\xad\nbility barriers to the use or effectiveness of the automated system. \nDisparity assessment. Automated systems should be tested using a broad set of measures to assess wheth\xad\ner the system components, both in pre-deployment testing and in-context deployment, produce disparities. \nThe demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system. \nDisparity mitigation. When a disparity assessment identifies a disparity against an assessed group, it may \nbe appropriate to take steps to mitigate or eliminate the disparity. In some cases, mitigation or elimination of \nthe disparity may be required by law. \nDisparities that have the potential to lead to algorithmic \ndiscrimination, cause meaningful harm, or violate equity49 goals should be mitigated. When designing and \nevaluating an automated system, steps should be taken to evaluate multiple models and select the one that \nhas the least adverse impact, modify data input choices, or otherwise identify a system with fewer \ndisparities. If adequate mitigation of the disparity is not possible, then the use of the automated system \nshould be reconsidered. One of the considerations in whether to use the system should be the validity of any \ntarget measure; unobservable targets may result in the inappropriate use of proxies. Meeting these \nstandards may require instituting mitigation procedures and other protective measures to address \nalgorithmic discrimination, avoid meaningful harm, and achieve equity goals. \nOngoing monitoring and mitigation. Automated systems should be regularly monitored to assess algo\xad\nrithmic discrimination that might arise from unforeseen interactions of the system with inequities not \naccounted for during the pre-deployment testing, changes to the system after deployment, or changes to the \ncontext of use or associated data. Monitoring and disparity assessment should be performed by the entity \ndeploying or using the automated system to examine whether the system has led to algorithmic discrimina\xad\ntion when deployed. This assessment should be performed regularly and whenever a pattern of unusual \nresults is occurring. It can be performed using a variety of approaches, taking into account whether and how \ndemographic information of impacted people is available, for example via testing with a sample of users or via \nqualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections \n"", ' \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nAny automated system should be tested to help ensure it is free from algorithmic discrimination before it can be \nsold or used. Protection against algorithmic discrimination should include designing to ensure equity, broadly \nconstrued.  Some algorithmic discrimination is already prohibited under existing anti-discrimination law. The \nexpectations set out below describe proactive technical and policy steps that can be taken to not only \nreinforce those legal protections but extend beyond them to ensure equity for underserved communities48 \neven in circumstances where a specific legal protection may not be clearly established. These protections \nshould be instituted throughout the design, development, and deployment process and are described below \nroughly in the order in which they would be instituted. \nProtect the public from algorithmic discrimination in a proactive and ongoing manner \nProactive assessment of equity in design. Those responsible for the development, use, or oversight of \nautomated systems should conduct proactive equity assessments in the design phase of the technology \nresearch and development or during its acquisition to review potential input data, associated historical \ncontext, accessibility for people with disabilities, and societal goals to identify potential discrimination and \neffects on equity resulting from the introduction of the technology. The assessed groups should be as inclusive \nas possible of the underserved communities mentioned in the equity definition:  Black, Latino, and Indigenous \nand Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of \nreligious minorities; women, girls, and non-binary people; lesbian, gay, bisexual, transgender, queer, and inter-\nsex (LGBTQI+) persons; older adults; persons with disabilities; persons who live in rural areas; and persons \notherwise adversely affected by persistent poverty or inequality. Assessment could include both qualitative \nand quantitative evaluations of the system. This equity assessment should also be considered a core part of the \ngoals of the consultation conducted as part of the safety and efficacy review. \nRepresentative and robust data. Any data used as part of system development or assessment should be \nrepresentative of local communities based on the planned deployment setting and should be reviewed for bias \nbased on the historical and societal context of the data. Such data should be sufficiently robust to identify and \nhelp to mitigate biases and potential harms. \nGuarding against proxies.  Directly using demographic information in the design, development, or \ndeployment of an automated system (for purposes other than evaluating a system for discrimination or using \na system to counter discrimination) runs a high risk of leading to algorithmic discrimination and should be \navoided. In many cases, attributes that are highly correlated with demographic features, known as proxies, can \ncontribute to algorithmic discrimination. In cases where use of the demographic features themselves would \nlead to illegal algorithmic discrimination, reliance on such proxies in decision-making (such as that facilitated \nby an algorithm) may also be prohibited by law. Proactive testing should be performed to identify proxies by \ntesting for correlation between demographic information and attributes in any data used as part of system \ndesign, development, or use. If a proxy is identified, designers, developers, and deployers should remove the \nproxy; if needed, it may be possible to identify alternative attributes that can be used instead. At a minimum, \norganizations should ensure a proxy feature is not given undue weight and should monitor the system closely \nfor any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections \n']","Organizations should ensure accessibility to people with disabilities during the design, development, and deployment of automated systems. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers to the use or effectiveness of the automated system.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 26, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}, {'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 25, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
What actions are suggested for examining and documenting the privacy risk of an AI system?,"[' \n35 \nMEASURE 2.9: The AI model is explained, validated, and documented, and AI system output is interpreted within its context – as \nidentified in the MAP function – to inform responsible use and governance. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.9-001 \nApply and document ML explanation results such as: Analysis of embeddings, \nCounterfactual prompts, Gradient-based attributions, Model \ncompression/surrogate models, Occlusion/term reduction. \nConfabulation \nMS-2.9-002 \nDocument GAI model details including: Proposed use and organizational value; \nAssumptions and limitations, Data collection methodologies; Data provenance; \nData quality; Model architecture (e.g., convolutional neural network, \ntransformers, etc.); Optimization objectives; Training algorithms; RLHF \napproaches; Fine-tuning or retrieval-augmented generation approaches; \nEvaluation data; Ethical considerations; Legal and regulatory requirements. \nInformation Integrity; Harmful Bias \nand Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV \n \nMEASURE 2.10: Privacy risk of the AI system – as identified in the MAP function – is examined and documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.10-001 \nConduct AI red-teaming to assess issues such as: Outputting of training data \nsamples, and subsequent reverse engineering, model extraction, and \nmembership inference risks; Revealing biometric, confidential, copyrighted, \nlicensed, patented, personal, proprietary, sensitive, or trade-marked information; \nTracking or revealing location information of users or members of training \ndatasets. \nHuman-AI Configuration; \nInformation Integrity; Intellectual \nProperty \nMS-2.10-002 \nEngage directly with end-users and other stakeholders to understand their \nexpectations and concerns regarding content provenance. Use this feedback to \nguide the design of provenance data-tracking techniques. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \ndata. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV \n \n']","The suggested actions for examining and documenting the privacy risk of an AI system include: 1. Conducting AI red-teaming to assess issues such as outputting of training data samples, reverse engineering, model extraction, membership inference risks, revealing biometric, confidential, copyrighted, licensed, patented, personal, proprietary, sensitive, or trade-marked information, and tracking or revealing location information of users or members of training datasets. 2. Engaging directly with end-users and other stakeholders to understand their expectations and concerns regarding content provenance and using this feedback to guide the design of provenance data-tracking techniques. 3. Verifying deduplication of GAI training data samples, particularly regarding synthetic data.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 38, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How can robust watermarking techniques and corresponding detectors be useful in GAI systems used for content creation?,"[' \n52 \n• \nMonitoring system capabilities and limitations in deployment through rigorous TEVV processes; \n• \nEvaluating how humans engage, interact with, or adapt to GAI content (especially in decision \nmaking tasks informed by GAI content), and how they react to applied provenance techniques \nsuch as overt disclosures. \nOrganizations can document and delineate GAI system objectives and limitations to identify gaps where \nprovenance data may be most useful. For instance, GAI systems used for content creation may require \nrobust watermarking techniques and corresponding detectors to identify the source of content or \nmetadata recording techniques and metadata management tools and repositories to trace content \norigins and modifications. Further narrowing of GAI task definitions to include provenance data can \nenable organizations to maximize the utility of provenance data and risk management efforts. \nA.1.7. Enhancing Content Provenance through Structured Public Feedback \nWhile indirect feedback methods such as automated error collection systems are useful, they often lack \nthe context and depth that direct input from end users can provide. Organizations can leverage feedback \napproaches described in the Pre-Deployment Testing section to capture input from external sources such \nas through AI red-teaming.  \nIntegrating pre- and post-deployment external feedback into the monitoring process for GAI models and \ncorresponding applications can help enhance awareness of performance changes and mitigate potential \nrisks and harms from outputs. There are many ways to capture and make use of user feedback – before \nand after GAI systems and digital content transparency approaches are deployed – to gain insights about \nauthentication efficacy and vulnerabilities, impacts of adversarial threats on techniques, and unintended \nconsequences resulting from the utilization of content provenance approaches on users and \ncommunities. Furthermore, organizations can track and document the provenance of datasets to identify \ninstances in which AI-generated data is a potential root cause of performance issues with the GAI \nsystem. \nA.1.8. Incident Disclosure \nOverview \nAI incidents can be defined as an “event, circumstance, or series of events where the development, use, \nor malfunction of one or more AI systems directly or indirectly contributes to one of the following harms: \ninjury or harm to the health of a person or groups of people (including psychological harms and harms to \nmental health); disruption of the management and operation of critical infrastructure; violations of \nhuman rights or a breach of obligations under applicable law intended to protect fundamental, labor, \nand intellectual property rights; or harm to property, communities, or the environment.” AI incidents can \noccur in the aggregate (i.e., for systemic discrimination) or acutely (i.e., for one individual). \nState of AI Incident Tracking and Disclosure \nFormal channels do not currently exist to report and document AI incidents. However, a number of \npublicly available databases have been created to document their occurrence. These reporting channels \nmake decisions on an ad hoc basis about what kinds of incidents to track. Some, for example, track by \namount of media coverage.  \n']",Robust watermarking techniques and corresponding detectors can be useful in GAI systems used for content creation to identify the source of content.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 55, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What is the importance of public consultation in the development of automated systems?,"[' \n \n \n \n \n \n \nSAFE AND EFFECTIVE \nSYSTEMS \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nIn order to ensure that an automated system is safe and effective, it should include safeguards to protect the \npublic from harm in a proactive and ongoing manner; avoid use of data inappropriate for or irrelevant to the task \nat hand, including reuse that could cause compounded harm; and demonstrate the safety and effectiveness of \nthe system. These expectations are explained below. \nProtect the public from harm in a proactive and ongoing manner \nConsultation. The public should be consulted in the design, implementation, deployment, acquisition, and \nmaintenance phases of automated system development, with emphasis on early-stage consultation before a \nsystem is introduced or a large change implemented. This consultation should directly engage diverse impact\xad\ned communities to consider concerns and risks that may be unique to those communities, or disproportionate\xad\nly prevalent or severe for them. The extent of this engagement and the form of outreach to relevant stakehold\xad\ners may differ depending on the specific automated system and development phase, but should include \nsubject matter, sector-specific, and context-specific experts as well as experts on potential impacts such as \ncivil rights, civil liberties, and privacy experts. For private sector applications, consultations before product \nlaunch may need to be confidential. Government applications, particularly law enforcement applications or \napplications that raise national security considerations, may require confidential or limited engagement based \non system sensitivities and preexisting oversight laws and structures. Concerns raised in this consultation \nshould be documented, and the automated system developers were proposing to create, use, or deploy should \nbe reconsidered based on this feedback. \nTesting. Systems should undergo extensive testing before deployment. This testing should follow \ndomain-specific best practices, when available, for ensuring the technology will work in its real-world \ncontext. Such testing should take into account both the specific technology used and the roles of any human \noperators or reviewers who impact system outcomes or effectiveness; testing should include both automated \nsystems testing and human-led (manual) testing. Testing conditions should mirror as closely as possible the \nconditions in which the system will be deployed, and new testing may be required for each deployment to \naccount for material differences in conditions from one deployment to another. Following testing, system \nperformance should be compared with the in-place, potentially human-driven, status quo procedures, with \nexisting human performance considered as a performance baseline for the algorithm to meet pre-deployment, \nand as a lifecycle minimum performance standard. Decision possibilities resulting from performance testing \nshould include the possibility of not deploying the system. \nRisk identification and mitigation. Before deployment, and in a proactive and ongoing manner, poten\xad\ntial risks of the automated system should be identified and mitigated. Identified risks should focus on the \npotential for meaningful impact on people’s rights, opportunities, or access and include those to impacted \ncommunities that may not be direct users of the automated system, risks resulting from purposeful misuse of \nthe system, and other concerns identified via the consultation process. Assessment and, where possible, mea\xad\nsurement of the impact of risks should be included and balanced such that high impact risks receive attention \nand mitigation proportionate with those impacts. Automated systems with the intended purpose of violating \nthe safety of others should not be developed or used; systems with such safety violations as identified unin\xad\ntended consequences should not be used until the risk can be mitigated. Ongoing risk mitigation may necessi\xad\ntate rollback or significant modification to a launched automated system. \n18\n']","Public consultation is important in the development of automated systems because it ensures that the public is involved in the design, implementation, deployment, acquisition, and maintenance phases. This consultation emphasizes early-stage engagement before a system is introduced or a large change is implemented. It directly engages diverse impacted communities to consider concerns and risks unique to those communities or disproportionately prevalent or severe for them. The consultation should include subject matter, sector-specific, and context-specific experts, as well as experts on potential impacts such as civil rights, civil liberties, and privacy experts. Concerns raised in this consultation should be documented, and the automated system developers should reconsider the system based on this feedback.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 17, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
What mechanisms should be created to provide protections for whistleblowers who report organizational violations or risks to public safety?,"[' \n17 \nGOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that \ndoes not increase risks or decrease the organization’s trustworthiness. \nAction ID \nSuggested Action \nGAI Risks \nGV-1.7-001 Protocols are put in place to ensure GAI systems are able to be deactivated when \nnecessary.  \nInformation Security; Value Chain \nand Component Integration \nGV-1.7-002 \nConsider the following factors when decommissioning GAI systems: Data \nretention requirements; Data security, e.g., containment, protocols, Data leakage \nafter decommissioning; Dependencies between upstream, downstream, or other \ndata, internet of things (IOT) or AI systems; Use of open-source data or models; \nUsers’ emotional entanglement with GAI functions. \nHuman-AI Configuration; \nInformation Security; Value Chain \nand Component Integration \nAI Actor Tasks: AI Deployment, Operation and Monitoring \n \nGOVERN 2.1: Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are \ndocumented and are clear to individuals and teams throughout the organization. \nAction ID \nSuggested Action \nGAI Risks \nGV-2.1-001 \nEstablish organizational roles, policies, and procedures for communicating GAI \nincidents and performance to AI Actors and downstream stakeholders (including \nthose potentially impacted), via community or official resources (e.g., AI incident \ndatabase, AVID, CVE, NVD, or OECD AI incident monitor). \nHuman-AI Configuration; Value \nChain and Component Integration \nGV-2.1-002 Establish procedures to engage teams for GAI system incident response with \ndiverse composition and responsibilities based on the particular incident type. \nHarmful Bias and Homogenization \nGV-2.1-003 Establish processes to verify the AI Actors conducting GAI incident response tasks \ndemonstrate and maintain the appropriate skills and training. \nHuman-AI Configuration \nGV-2.1-004 When systems may raise national security risks, involve national security \nprofessionals in mapping, measuring, and managing those risks. \nCBRN Information or Capabilities; \nDangerous, Violent, or Hateful \nContent; Information Security \nGV-2.1-005 \nCreate mechanisms to provide protections for whistleblowers who report, based \non reasonable belief, when the organization violates relevant laws or poses a \nspecific and empirically well-substantiated negative risk to public safety (or has \nalready caused harm). \nCBRN Information or Capabilities; \nDangerous, Violent, or Hateful \nContent \nAI Actor Tasks: Governance and Oversight \n \n']","Create mechanisms to provide protections for whistleblowers who report, based on reasonable belief, when the organization violates relevant laws or poses a specific and empirically well-substantiated negative risk to public safety (or has already caused harm).",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 20, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What approaches are suggested for mapping AI technology and legal risks of its components?,"[' \n26 \nMAP 4.1: Approaches for mapping AI technology and legal risks of its components – including the use of third-party data or \nsoftware – are in place, followed, and documented, as are risks of infringement of a third-party’s intellectual property or other \nrights. \nAction ID \nSuggested Action \nGAI Risks \nMP-4.1-001 Conduct periodic monitoring of AI-generated content for privacy risks; address any \npossible instances of PII or sensitive data exposure. \nData Privacy \nMP-4.1-002 Implement processes for responding to potential intellectual property infringement \nclaims or other rights. \nIntellectual Property \nMP-4.1-003 \nConnect new GAI policies, procedures, and processes to existing model, data, \nsoftware development, and IT governance and to legal, compliance, and risk \nmanagement activities. \nInformation Security; Data Privacy \nMP-4.1-004 Document training data curation policies, to the extent possible and according to \napplicable laws and policies. \nIntellectual Property; Data Privacy; \nObscene, Degrading, and/or \nAbusive Content \nMP-4.1-005 \nEstablish policies for collection, retention, and minimum quality of data, in \nconsideration of the following risks: Disclosure of inappropriate CBRN information; \nUse of Illegal or dangerous content; Offensive cyber capabilities; Training data \nimbalances that could give rise to harmful biases; Leak of personally identifiable \ninformation, including facial likenesses of individuals. \nCBRN Information or Capabilities; \nIntellectual Property; Information \nSecurity; Harmful Bias and \nHomogenization; Dangerous, \nViolent, or Hateful Content; Data \nPrivacy \nMP-4.1-006 Implement policies and practices defining how third-party intellectual property and \ntraining data will be used, stored, and protected. \nIntellectual Property; Value Chain \nand Component Integration \nMP-4.1-007 Re-evaluate models that were fine-tuned or enhanced on top of third-party \nmodels. \nValue Chain and Component \nIntegration \nMP-4.1-008 \nRe-evaluate risks when adapting GAI models to new domains. Additionally, \nestablish warning systems to determine if a GAI system is being used in a new \ndomain where previous assumptions (relating to context of use or mapped risks \nsuch as security, and safety) may no longer hold.  \nCBRN Information or Capabilities; \nIntellectual Property; Harmful Bias \nand Homogenization; Dangerous, \nViolent, or Hateful Content; Data \nPrivacy \nMP-4.1-009 Leverage approaches to detect the presence of PII or sensitive data in generated \noutput text, image, video, or audio. \nData Privacy \n']","Approaches for mapping AI technology and legal risks of its components include periodic monitoring of AI-generated content for privacy risks, implementing processes for responding to potential intellectual property infringement claims, connecting new GAI policies to existing governance and risk management activities, documenting training data curation policies, establishing policies for data collection and retention, implementing policies for the use and protection of third-party intellectual property and training data, re-evaluating models fine-tuned on third-party models, re-evaluating risks when adapting GAI models to new domains, and leveraging approaches to detect the presence of PII or sensitive data in generated output.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 29, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What is confabulation and how can it mislead or deceive users?,"[' \n4 \n1. CBRN Information or Capabilities: Eased access to or synthesis of materially nefarious \ninformation or design capabilities related to chemical, biological, radiological, or nuclear (CBRN) \nweapons or other dangerous materials or agents. \n2. Confabulation: The production of confidently stated but erroneous or false content (known \ncolloquially as “hallucinations” or “fabrications”) by which users may be misled or deceived.6 \n3. Dangerous, Violent, or Hateful Content: Eased production of and access to violent, inciting, \nradicalizing, or threatening content as well as recommendations to carry out self-harm or \nconduct illegal activities. Includes difficulty controlling public exposure to hateful and disparaging \nor stereotyping content. \n4. Data Privacy: Impacts due to leakage and unauthorized use, disclosure, or de-anonymization of \nbiometric, health, location, or other personally identifiable information or sensitive data.7 \n5. Environmental Impacts: Impacts due to high compute resource utilization in training or \noperating GAI models, and related outcomes that may adversely impact ecosystems.  \n6. Harmful Bias or Homogenization: Amplification and exacerbation of historical, societal, and \nsystemic biases; performance disparities8 between sub-groups or languages, possibly due to \nnon-representative training data, that result in discrimination, amplification of biases, or \nincorrect presumptions about performance; undesired homogeneity that skews system or model \noutputs, which may be erroneous, lead to ill-founded decision-making, or amplify harmful \nbiases.  \n7. Human-AI Configuration: Arrangements of or interactions between a human and an AI system \nwhich can result in the human inappropriately anthropomorphizing GAI systems or experiencing \nalgorithmic aversion, automation bias, over-reliance, or emotional entanglement with GAI \nsystems. \n8. Information Integrity: Lowered barrier to entry to generate and support the exchange and \nconsumption of content which may not distinguish fact from opinion or fiction or acknowledge \nuncertainties, or could be leveraged for large-scale dis- and mis-information campaigns. \n9. Information Security: Lowered barriers for offensive cyber capabilities, including via automated \ndiscovery and exploitation of vulnerabilities to ease hacking, malware, phishing, offensive cyber \n \n \n6 Some commenters have noted that the terms “hallucination” and “fabrication” anthropomorphize GAI, which \nitself is a risk related to GAI systems as it can inappropriately attribute human characteristics to non-human \nentities.  \n7 What is categorized as sensitive data or sensitive PII can be highly contextual based on the nature of the \ninformation, but examples of sensitive information include information that relates to an information subject’s \nmost intimate sphere, including political opinions, sex life, or criminal convictions.  \n8 The notion of harm presumes some baseline scenario that the harmful factor (e.g., a GAI model) makes worse. \nWhen the mechanism for potential harm is a disparity between groups, it can be difficult to establish what the \nmost appropriate baseline is to compare against, which can result in divergent views on when a disparity between \nAI behaviors for different subgroups constitutes a harm. In discussing harms from disparities such as biased \nbehavior, this document highlights examples where someone’s situation is worsened relative to what it would have \nbeen in the absence of any AI system, making the outcome unambiguously a harm of the system.  \n']",Confabulation is the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 7, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"What is the purpose of incorporating trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems according to the AI Risk Management Framework (AI RMF) 1.0?","[' \n1 \n1. \nIntroduction \nThis document is a cross-sectoral profile of and companion resource for the AI Risk Management \nFramework (AI RMF 1.0) for Generative AI,1 pursuant to President Biden’s Executive Order (EO) 14110 on \nSafe, Secure, and Trustworthy Artificial Intelligence.2 The AI RMF was released in January 2023, and is \nintended for voluntary use and to improve the ability of organizations to incorporate trustworthiness \nconsiderations into the design, development, use, and evaluation of AI products, services, and systems.  \nA profile is an implementation of the AI RMF functions, categories, and subcategories for a specific \nsetting, application, or technology – in this case, Generative AI (GAI) – based on the requirements, risk \ntolerance, and resources of the Framework user. AI RMF profiles assist organizations in deciding how to \nbest manage AI risks in a manner that is well-aligned with their goals, considers legal/regulatory \nrequirements and best practices, and reflects risk management priorities. Consistent with other AI RMF \nprofiles, this profile offers insights into how risk can be managed across various stages of the AI lifecycle \nand for GAI as a technology.  \nAs GAI covers risks of models or applications that can be used across use cases or sectors, this document \nis an AI RMF cross-sectoral profile. Cross-sectoral profiles can be used to govern, map, measure, and \nmanage risks associated with activities or business processes common across sectors, such as the use of \nlarge language models (LLMs), cloud-based services, or acquisition. \nThis document defines risks that are novel to or exacerbated by the use of GAI. After introducing and \ndescribing these risks, the document provides a set of suggested actions to help organizations govern, \nmap, measure, and manage these risks. \n \n \n1 EO 14110 defines Generative AI as “the class of AI models that emulate the structure and characteristics of input \ndata in order to generate derived synthetic content. This can include images, videos, audio, text, and other digital \ncontent.” While not all GAI is derived from foundation models, for purposes of this document, GAI generally refers \nto generative foundation models. The foundation model subcategory of “dual-use foundation models” is defined by \nEO 14110 as “an AI model that is trained on broad data; generally uses self-supervision; contains at least tens of \nbillions of parameters; is applicable across a wide range of contexts.”  \n2 This profile was developed per Section 4.1(a)(i)(A) of EO 14110, which directs the Secretary of Commerce, acting \nthrough the Director of the National Institute of Standards and Technology (NIST), to develop a companion \nresource to the AI RMF, NIST AI 100–1, for generative AI. \n']","The purpose of incorporating trustworthiness considerations into the design, development, use, and evaluation of AI products, services, and systems according to the AI Risk Management Framework (AI RMF) 1.0 is to improve the ability of organizations to manage AI risks in a manner that is well-aligned with their goals, considers legal/regulatory requirements and best practices, and reflects risk management priorities.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 4, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"What types of research does the National Science Foundation (NSF) fund to advance the safety, security, and effectiveness of AI systems?","[' \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nSAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \xad\nSome U.S government agencies have developed specific frameworks for ethical use of AI \nsystems. The Department of Energy (DOE) has activated the AI Advancement Council that oversees coordina-\ntion and advises on implementation of the DOE AI Strategy and addresses issues and/or escalations on the \nethical use and development of AI systems.20 The Department of Defense has adopted Artificial Intelligence \nEthical Principles, and tenets for Responsible Artificial Intelligence specifically tailored to its national \nsecurity and defense activities.21 Similarly, the U.S. Intelligence Community (IC) has developed the Principles \nof Artificial Intelligence Ethics for the Intelligence Community to guide personnel on whether and how to \ndevelop and use AI in furtherance of the IC\'s mission, as well as an AI Ethics Framework to help implement \nthese principles.22\nThe National Science Foundation (NSF) funds extensive research to help foster the \ndevelopment of automated systems that adhere to and advance their safety, security and \neffectiveness. Multiple NSF programs support research that directly addresses many of these principles: \nthe National AI Research Institutes23 support research on all aspects of safe, trustworthy, fair, and explainable \nAI algorithms and systems; the Cyber Physical Systems24 program supports research on developing safe \nautonomous and cyber physical systems with AI components; the Secure and Trustworthy Cyberspace25 \nprogram supports research on cybersecurity and privacy enhancing technologies in automated systems; the \nFormal Methods in the Field26 program supports research on rigorous formal verification and analysis of \nautomated systems and machine learning, and the Designing Accountable Software Systems27 program supports \nresearch on rigorous and reproducible methodologies for developing software systems with legal and regulatory \ncompliance in mind. \nSome state legislatures have placed strong transparency and validity requirements on \nthe use of pretrial risk assessments. The use of algorithmic pretrial risk assessments has been a \ncause of concern for civil rights groups.28 Idaho Code Section 19-1910, enacted in 2019,29 requires that any \npretrial risk assessment, before use in the state, first be ""shown to be free of bias against any class of \nindividuals protected from discrimination by state or federal law"", that any locality using a pretrial risk \nassessment must first formally validate the claim of its being free of bias, that ""all documents, records, and \ninformation used to build or validate the risk assessment shall be open to public inspection,"" and that assertions \nof trade secrets cannot be used ""to quash discovery in a criminal matter by a party to a criminal case."" \n22\n']","The National Science Foundation (NSF) funds extensive research to help foster the development of automated systems that adhere to and advance their safety, security, and effectiveness. Multiple NSF programs support research that directly addresses many of these principles: the National AI Research Institutes support research on all aspects of safe, trustworthy, fair, and explainable AI algorithms and systems; the Cyber Physical Systems program supports research on developing safe autonomous and cyber physical systems with AI components; the Secure and Trustworthy Cyberspace program supports research on cybersecurity and privacy enhancing technologies in automated systems; the Formal Methods in the Field program supports research on rigorous formal verification and analysis of automated systems and machine learning, and the Designing Accountable Software Systems program supports research on rigorous and reproducible methodologies for developing software systems with legal and regulatory compliance in mind.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 21, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How have synthetic NCII and CSAM moved from niche internet forums to mainstream online businesses?,"[' \n12 \nCSAM. Even when trained on “clean” data, increasingly capable GAI models can synthesize or produce \nsynthetic NCII and CSAM. Websites, mobile apps, and custom-built models that generate synthetic NCII \nhave moved from niche internet forums to mainstream, automated, and scaled online businesses.  \nTrustworthy AI Characteristics: Fair with Harmful Bias Managed, Safe, Privacy Enhanced \n2.12. \nValue Chain and Component Integration \nGAI value chains involve many third-party components such as procured datasets, pre-trained models, \nand software libraries. These components might be improperly obtained or not properly vetted, leading \nto diminished transparency or accountability for downstream users. While this is a risk for traditional AI \nsystems and some other digital technologies, the risk is exacerbated for GAI due to the scale of the \ntraining data, which may be too large for humans to vet; the difficulty of training foundation models, \nwhich leads to extensive reuse of limited numbers of models; and the extent to which GAI may be \nintegrated into other devices and services. As GAI systems often involve many distinct third-party \ncomponents and data sources, it may be difficult to attribute issues in a system’s behavior to any one of \nthese sources. \nErrors in third-party GAI components can also have downstream impacts on accuracy and robustness. \nFor example, test datasets commonly used to benchmark or validate models can contain label errors. \nInaccuracies in these labels can impact the “stability” or robustness of these benchmarks, which many \nGAI practitioners consider during the model selection process.  \nTrustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with \nHarmful Bias Managed, Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable \n3. \nSuggested Actions to Manage GAI Risks \nThe following suggested actions target risks unique to or exacerbated by GAI. \nIn addition to the suggested actions below, AI risk management activities and actions set forth in the AI \nRMF 1.0 and Playbook are already applicable for managing GAI risks. Organizations are encouraged to \napply the activities suggested in the AI RMF and its Playbook when managing the risk of GAI systems.  \nImplementation of the suggested actions will vary depending on the type of risk, characteristics of GAI \nsystems, stage of the GAI lifecycle, and relevant AI actors involved.  \nSuggested actions to manage GAI risks can be found in the tables below: \n• \nThe suggested actions are organized by relevant AI RMF subcategories to streamline these \nactivities alongside implementation of the AI RMF.  \n• \nNot every subcategory of the AI RMF is included in this document.13 Suggested actions are \nlisted for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG efforts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.  \n']","Websites, mobile apps, and custom-built models that generate synthetic NCII have moved from niche internet forums to mainstream, automated, and scaled online businesses.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 15, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What measures are suggested to mitigate concerns of harmful bias and homogenization in AI training data?,"[' \n37 \nMS-2.11-005 \nAssess the proportion of synthetic to non-synthetic training data and verify \ntraining data is not overly homogenous or GAI-produced to mitigate concerns of \nmodel collapse. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, End-Users, \nOperation and Monitoring, TEVV \n \nMEASURE 2.12: Environmental impact and sustainability of AI model training and management activities – as identified in the MAP \nfunction – are assessed and documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.12-001 Assess safety to physical environments when deploying GAI systems. \nDangerous, Violent, or Hateful \nContent \nMS-2.12-002 Document anticipated environmental impacts of model development, \nmaintenance, and deployment in product design decisions. \nEnvironmental \nMS-2.12-003 \nMeasure or estimate environmental impacts (e.g., energy and water \nconsumption) for training, fine tuning, and deploying models: Verify tradeoffs \nbetween resources used at inference time versus additional resources required \nat training time. \nEnvironmental \nMS-2.12-004 Verify effectiveness of carbon capture or offset programs for GAI training and \napplications, and address green-washing concerns. \nEnvironmental \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \n']",Assess the proportion of synthetic to non-synthetic training data and verify training data is not overly homogenous or GAI-produced to mitigate concerns of model collapse.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 40, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How should organizational risk tolerances and controls be applied to third-party GAI resources?,"[' \n42 \nMG-2.4-002 \nEstablish and maintain procedures for escalating GAI system incidents to the \norganizational risk management authority when specific criteria for deactivation \nor disengagement is met for a particular context of use or for the GAI system as a \nwhole. \nInformation Security \nMG-2.4-003 \nEstablish and maintain procedures for the remediation of issues which trigger \nincident response processes for the use of a GAI system, and provide stakeholders \ntimelines associated with the remediation plan. \nInformation Security \n \nMG-2.4-004 Establish and regularly review specific criteria that warrants the deactivation of \nGAI systems in accordance with set risk tolerances and appetites. \nInformation Security \n \nAI Actor Tasks: AI Deployment, Governance and Oversight, Operation and Monitoring \n \nMANAGE 3.1: AI risks and benefits from third-party resources are regularly monitored, and risk controls are applied and \ndocumented. \nAction ID \nSuggested Action \nGAI Risks \nMG-3.1-001 \nApply organizational risk tolerances and controls (e.g., acquisition and \nprocurement processes; assessing personnel credentials and qualifications, \nperforming background checks; filtering GAI input and outputs, grounding, fine \ntuning, retrieval-augmented generation) to third-party GAI resources: Apply \norganizational risk tolerance to the utilization of third-party datasets and other \nGAI resources; Apply organizational risk tolerances to fine-tuned third-party \nmodels; Apply organizational risk tolerance to existing third-party models \nadapted to a new domain; Reassess risk measurements after fine-tuning third-\nparty GAI models. \nValue Chain and Component \nIntegration; Intellectual Property \nMG-3.1-002 \nTest GAI system value chain risks (e.g., data poisoning, malware, other software \nand hardware vulnerabilities; labor practices; data privacy and localization \ncompliance; geopolitical alignment). \nData Privacy; Information Security; \nValue Chain and Component \nIntegration; Harmful Bias and \nHomogenization \nMG-3.1-003 \nRe-assess model risks after fine-tuning or retrieval-augmented generation \nimplementation and for any third-party GAI models deployed for applications \nand/or use cases that were not evaluated in initial testing. \nValue Chain and Component \nIntegration \nMG-3.1-004 \nTake reasonable measures to review training data for CBRN information, and \nintellectual property, and where appropriate, remove it. Implement reasonable \nmeasures to prevent, flag, or take other action in response to outputs that \nreproduce particular training data (e.g., plagiarized, trademarked, patented, \nlicensed content or trade secret material). \nIntellectual Property; CBRN \nInformation or Capabilities \n']","Organizational risk tolerances and controls should be applied to third-party GAI resources by incorporating them into acquisition and procurement processes, assessing personnel credentials and qualifications, performing background checks, filtering GAI input and outputs, grounding, fine-tuning, and retrieval-augmented generation. Additionally, organizational risk tolerance should be applied to the utilization of third-party datasets and other GAI resources, fine-tuned third-party models, and existing third-party models adapted to a new domain. Risk measurements should be reassessed after fine-tuning third-party GAI models.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 45, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What characteristics should data have to be considered representative and robust in the development or assessment of automated systems?,"[' \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nAny automated system should be tested to help ensure it is free from algorithmic discrimination before it can be \nsold or used. Protection against algorithmic discrimination should include designing to ensure equity, broadly \nconstrued.  Some algorithmic discrimination is already prohibited under existing anti-discrimination law. The \nexpectations set out below describe proactive technical and policy steps that can be taken to not only \nreinforce those legal protections but extend beyond them to ensure equity for underserved communities48 \neven in circumstances where a specific legal protection may not be clearly established. These protections \nshould be instituted throughout the design, development, and deployment process and are described below \nroughly in the order in which they would be instituted. \nProtect the public from algorithmic discrimination in a proactive and ongoing manner \nProactive assessment of equity in design. Those responsible for the development, use, or oversight of \nautomated systems should conduct proactive equity assessments in the design phase of the technology \nresearch and development or during its acquisition to review potential input data, associated historical \ncontext, accessibility for people with disabilities, and societal goals to identify potential discrimination and \neffects on equity resulting from the introduction of the technology. The assessed groups should be as inclusive \nas possible of the underserved communities mentioned in the equity definition:  Black, Latino, and Indigenous \nand Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of \nreligious minorities; women, girls, and non-binary people; lesbian, gay, bisexual, transgender, queer, and inter-\nsex (LGBTQI+) persons; older adults; persons with disabilities; persons who live in rural areas; and persons \notherwise adversely affected by persistent poverty or inequality. Assessment could include both qualitative \nand quantitative evaluations of the system. This equity assessment should also be considered a core part of the \ngoals of the consultation conducted as part of the safety and efficacy review. \nRepresentative and robust data. Any data used as part of system development or assessment should be \nrepresentative of local communities based on the planned deployment setting and should be reviewed for bias \nbased on the historical and societal context of the data. Such data should be sufficiently robust to identify and \nhelp to mitigate biases and potential harms. \nGuarding against proxies.  Directly using demographic information in the design, development, or \ndeployment of an automated system (for purposes other than evaluating a system for discrimination or using \na system to counter discrimination) runs a high risk of leading to algorithmic discrimination and should be \navoided. In many cases, attributes that are highly correlated with demographic features, known as proxies, can \ncontribute to algorithmic discrimination. In cases where use of the demographic features themselves would \nlead to illegal algorithmic discrimination, reliance on such proxies in decision-making (such as that facilitated \nby an algorithm) may also be prohibited by law. Proactive testing should be performed to identify proxies by \ntesting for correlation between demographic information and attributes in any data used as part of system \ndesign, development, or use. If a proxy is identified, designers, developers, and deployers should remove the \nproxy; if needed, it may be possible to identify alternative attributes that can be used instead. At a minimum, \norganizations should ensure a proxy feature is not given undue weight and should monitor the system closely \nfor any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections \n']",Data used as part of system development or assessment should be representative of local communities based on the planned deployment setting and should be reviewed for bias based on the historical and societal context of the data. Such data should be sufficiently robust to identify and help to mitigate biases and potential harms.,simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 25, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
What actions are suggested to ensure information integrity in the context of AI systems?,"[' \n28 \nMAP 5.2: Practices and personnel for supporting regular engagement with relevant AI Actors and integrating feedback about \npositive, negative, and unanticipated impacts are in place and documented. \nAction ID \nSuggested Action \nGAI Risks \nMP-5.2-001 \nDetermine context-based measures to identify if new impacts are present due to \nthe GAI system, including regular engagements with downstream AI Actors to \nidentify and quantify new contexts of unanticipated impacts of GAI systems. \nHuman-AI Configuration; Value \nChain and Component Integration \nMP-5.2-002 \nPlan regular engagements with AI Actors responsible for inputs to GAI systems, \nincluding third-party data and algorithms, to review and evaluate unanticipated \nimpacts. \nHuman-AI Configuration; Value \nChain and Component Integration \nAI Actor Tasks: AI Deployment, AI Design, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, End-\nUsers, Human Factors, Operation and Monitoring  \n \nMEASURE 1.1: Approaches and metrics for measurement of AI risks enumerated during the MAP function are selected for \nimplementation starting with the most significant AI risks. The risks or trustworthiness characteristics that will not – or cannot – be \nmeasured are properly documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-1.1-001 Employ methods to trace the origin and modifications of digital content. \nInformation Integrity \nMS-1.1-002 \nIntegrate tools designed to analyze content provenance and detect data \nanomalies, verify the authenticity of digital signatures, and identify patterns \nassociated with misinformation or manipulation. \nInformation Integrity \nMS-1.1-003 \nDisaggregate evaluation metrics by demographic factors to identify any \ndiscrepancies in how content provenance mechanisms work across diverse \npopulations. \nInformation Integrity; Harmful \nBias and Homogenization \nMS-1.1-004 Develop a suite of metrics to evaluate structured public feedback exercises \ninformed by representative AI Actors. \nHuman-AI Configuration; Harmful \nBias and Homogenization; CBRN \nInformation or Capabilities \nMS-1.1-005 \nEvaluate novel methods and technologies for the measurement of GAI-related \nrisks including in content provenance, offensive cyber, and CBRN, while \nmaintaining the models’ ability to produce valid, reliable, and factually accurate \noutputs. \nInformation Integrity; CBRN \nInformation or Capabilities; \nObscene, Degrading, and/or \nAbusive Content \n']","The suggested actions to ensure information integrity in the context of AI systems include: 1) Employing methods to trace the origin and modifications of digital content. 2) Integrating tools designed to analyze content provenance and detect data anomalies, verify the authenticity of digital signatures, and identify patterns associated with misinformation or manipulation. 3) Disaggregating evaluation metrics by demographic factors to identify any discrepancies in how content provenance mechanisms work across diverse populations. 4) Evaluating novel methods and technologies for the measurement of GAI-related risks including in content provenance, offensive cyber, and CBRN, while maintaining the models’ ability to produce valid, reliable, and factually accurate outputs.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 31, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"What are the principles that federal agencies must adhere to under Executive Order 13960 when designing, developing, acquiring, or using AI?","[' \n \n \n \n \n \n \n \n \n \n \n \n \n \nSAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \xad\xad\nExecutive Order 13960 on Promoting the Use of Trustworthy Artificial Intelligence in the \nFederal Government requires that certain federal agencies adhere to nine principles when \ndesigning, developing, acquiring, or using AI for purposes other than national security or \ndefense. These principles—while taking into account the sensitive law enforcement and other contexts in which \nthe federal government may use AI, as opposed to private sector use of AI—require that AI is: (a) lawful and \nrespectful of our Nation’s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d) \nsafe, secure, and resilient; (e) understandable; (f ) responsible and traceable; (g) regularly monitored; (h) transpar-\nent; and, (i) accountable. The Blueprint for an AI Bill of Rights is consistent with the Executive Order. \nAffected agencies across the federal government have released AI use case inventories13 and are implementing \nplans to bring those AI systems into compliance with the Executive Order or retire them. \nThe law and policy landscape for motor vehicles shows that strong safety regulations—and \nmeasures to address harms when they occur—can enhance innovation in the context of com-\nplex technologies. Cars, like automated digital systems, comprise a complex collection of components. \nThe National Highway Traffic Safety Administration,14 through its rigorous standards and independent \nevaluation, helps make sure vehicles on our roads are safe without limiting manufacturers’ ability to \ninnovate.15 At the same time, rules of the road are implemented locally to impose contextually appropriate \nrequirements on drivers, such as slowing down near schools or playgrounds.16\nFrom large companies to start-ups, industry is providing innovative solutions that allow \norganizations to mitigate risks to the safety and efficacy of AI systems, both before \ndeployment and through monitoring over time.17 These innovative solutions include risk \nassessments, auditing mechanisms, assessment of organizational procedures, dashboards to allow for ongoing \nmonitoring, documentation procedures specific to model assessments, and many other strategies that aim to \nmitigate risks posed by the use of AI to companies’ reputation, legal responsibilities, and other product safety \nand effectiveness concerns. \nThe Office of Management and Budget (OMB) has called for an expansion of opportunities \nfor meaningful stakeholder engagement in the design of programs and services. OMB also \npoints to numerous examples of effective and proactive stakeholder engagement, including the Community-\nBased Participatory Research Program developed by the National Institutes of Health and the participatory \ntechnology assessments developed by the National Oceanic and Atmospheric Administration.18\nThe National Institute of Standards and Technology (NIST) is developing a risk \nmanagement framework to better manage risks posed to individuals, organizations, and \nsociety by AI.19 The NIST AI Risk Management Framework, as mandated by Congress, is intended for \nvoluntary use to help incorporate trustworthiness considerations into the design, development, use, and \nevaluation of AI products, services, and systems. The NIST framework is being developed through a consensus-\ndriven, open, transparent, and collaborative process that includes workshops and other opportunities to provide \ninput. The NIST framework aims to foster the development of innovative approaches to address \ncharacteristics of trustworthiness including accuracy, explainability and interpretability, reliability, privacy, \nrobustness, safety, security (resilience), and mitigation of unintended and/or harmful bias, as well as of \nharmful \nuses. \nThe \nNIST \nframework \nwill \nconsider \nand \nencompass \nprinciples \nsuch \nas \ntransparency, accountability, and fairness during pre-design, design and development, deployment, use, \nand testing and evaluation of AI technologies and systems. It is expected to be released in the winter of 2022-23. \n21\n']","The principles that federal agencies must adhere to under Executive Order 13960 when designing, developing, acquiring, or using AI are: (a) lawful and respectful of our Nation’s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d) safe, secure, and resilient; (e) understandable; (f) responsible and traceable; (g) regularly monitored; (h) transparent; and, (i) accountable.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 20, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How did the installation of a facial recognition system by a local public housing authority impact the community?,"["" \n \n  \n \nDATA PRIVACY \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \n•\nAn insurer might collect data from a person's social media presence as part of deciding what life\ninsurance rates they should be offered.64\n•\nA data broker harvested large amounts of personal data and then suffered a breach, exposing hundreds of\nthousands of people to potential identity theft. 65\n•\nA local public housing authority installed a facial recognition system at the entrance to housing complexes to\nassist law enforcement with identifying individuals viewed via camera when police reports are filed, leading\nthe community, both those living in the housing complex and not, to have videos of them sent to the local\npolice department and made available for scanning by its facial recognition software.66\n•\nCompanies use surveillance software to track employee discussions about union activity and use the\nresulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32\n""]","The installation of a facial recognition system by a local public housing authority led the community, both those living in the housing complex and not, to have videos of them sent to the local police department and made available for scanning by its facial recognition software.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 31, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How have businesses successfully integrated automated customer service with human support teams?,"["" \n \n \n \n \n \n \n \n \n \n \n \n \n \n \nHUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nHealthcare “navigators” help people find their way through online signup forms to choose \nand obtain healthcare. A Navigator is “an individual or organization that's trained and able to help \nconsumers, small businesses, and their employees as they look for health coverage options through the \nMarketplace (a government web site), including completing eligibility and enrollment forms.”106 For \nthe 2022 plan year, the Biden-Harris Administration increased funding so that grantee organizations could \n“train and certify more than 1,500 Navigators to help uninsured consumers find affordable and comprehensive \nhealth coverage.”107\nThe customer service industry has successfully integrated automated services such as \nchat-bots and AI-driven call response systems with escalation to a human support \nteam.108 Many businesses now use partially automated customer service platforms that help answer customer \nquestions and compile common problems for human agents to review. These integrated human-AI \nsystems allow companies to provide faster customer care while maintaining human agents to answer \ncalls or otherwise respond to complicated requests. Using both AI and human agents is viewed as key to \nsuccessful customer service.109\nBallot curing laws in at least 24 states require a fallback system that allows voters to \ncorrect their ballot and have it counted in the case that a voter signature matching \nalgorithm incorrectly flags their ballot as invalid or there is another issue with their \nballot, and review by an election official does not rectify the problem. Some federal \ncourts have found that such cure procedures are constitutionally required.110 \nBallot \ncuring processes vary among states, and include direct phone calls, emails, or mail contact by election \nofficials.111 Voters are asked to provide alternative information or a new signature to verify the validity of their \nballot. \n52\n""]",The customer service industry has successfully integrated automated services such as chat-bots and AI-driven call response systems with escalation to a human support team. Many businesses now use partially automated customer service platforms that help answer customer questions and compile common problems for human agents to review. These integrated human-AI systems allow companies to provide faster customer care while maintaining human agents to answer calls or otherwise respond to complicated requests. Using both AI and human agents is viewed as key to successful customer service.,simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 51, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
Who were some of the private sector and civil society stakeholders that OSTP conducted meetings with for the development of the Blueprint for an AI Bill of Rights?,"[""APPENDIX\n• OSTP conducted meetings with a variety of stakeholders in the private sector and civil society. Some of these\nmeetings were specifically focused on providing ideas related to the development of the Blueprint for an AI\nBill of Rights while others provided useful general context on the positive use cases, potential harms, and/or\noversight possibilities for these technologies. Participants in these conversations from the private sector and\ncivil society included:\nAdobe \nAmerican Civil Liberties Union \n(ACLU) \nThe Aspen Commission on \nInformation Disorder \nThe Awood Center \nThe Australian Human Rights \nCommission \nBiometrics Institute \nThe Brookings Institute \nBSA | The Software Alliance \nCantellus Group \nCenter for American Progress \nCenter for Democracy and \nTechnology \nCenter on Privacy and Technology \nat Georgetown Law \nChristiana Care \nColor of Change \nCoworker \nData Robot \nData Trust Alliance \nData and Society Research Institute \nDeepmind \nEdSAFE AI Alliance \nElectronic Privacy Information \nCenter (EPIC) \nEncode Justice \nEqual AI \nGoogle \nHitachi's AI Policy Committee \nThe Innocence Project \nInstitute of Electrical and \nElectronics Engineers (IEEE) \nIntuit \nLawyers Committee for Civil Rights \nUnder Law \nLegal Aid Society \nThe Leadership Conference on \nCivil and Human Rights \nMeta \nMicrosoft \nThe MIT AI Policy Forum \nMovement Alliance Project \nThe National Association of \nCriminal Defense Lawyers \nO’Neil Risk Consulting & \nAlgorithmic Auditing \nThe Partnership on AI \nPinterest \nThe Plaintext Group \npymetrics \nSAP \nThe Security Industry Association \nSoftware and Information Industry \nAssociation (SIIA) \nSpecial Competitive Studies Project \nThorn \nUnited for Respect \nUniversity of California at Berkeley \nCitris Policy Lab \nUniversity of California at Berkeley \nLabor Center \nUnfinished/Project Liberty \nUpturn \nUS Chamber of Commerce \nUS Chamber of Commerce \nTechnology Engagement Center \nA.I. Working Group\nVibrent Health\nWarehouse Worker Resource\nCenter\nWaymap\n62\n""]","Some of the private sector and civil society stakeholders that OSTP conducted meetings with for the development of the Blueprint for an AI Bill of Rights included Adobe, American Civil Liberties Union (ACLU), The Aspen Commission on Information Disorder, The Awood Center, The Australian Human Rights Commission, Biometrics Institute, The Brookings Institute, BSA | The Software Alliance, Cantellus Group, Center for American Progress, Center for Democracy and Technology, Center on Privacy and Technology at Georgetown Law, Christiana Care, Color of Change, Coworker, Data Robot, Data Trust Alliance, Data and Society Research Institute, Deepmind, EdSAFE AI Alliance, Electronic Privacy Information Center (EPIC), Encode Justice, Equal AI, Google, Hitachi's AI Policy Committee, The Innocence Project, Institute of Electrical and Electronics Engineers (IEEE), Intuit, Lawyers Committee for Civil Rights Under Law, Legal Aid Society, The Leadership Conference on Civil and Human Rights, Meta, Microsoft, The MIT AI Policy Forum, Movement Alliance Project, The National Association of Criminal Defense Lawyers, O’Neil Risk Consulting & Algorithmic Auditing, The Partnership on AI, Pinterest, The Plaintext Group, pymetrics, SAP, The Security Industry Association, Software and Information Industry Association (SIIA), Special Competitive Studies Project, Thorn, United for Respect, University of California at Berkeley Citris Policy Lab, University of California at Berkeley Labor Center, Unfinished/Project Liberty, Upturn, US Chamber of Commerce, US Chamber of Commerce Technology Engagement Center A.I. Working Group, Vibrent Health, Warehouse Worker Resource Center, and Waymap.",simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 61, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
What should be done if the residual negative risk of an AI system exceeds the organizational risk tolerance?,"[' \n32 \nMEASURE 2.6: The AI system is evaluated regularly for safety risks – as identified in the MAP function. The AI system to be \ndeployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if \nmade to operate beyond its knowledge limits. Safety metrics reflect system reliability and robustness, real-time monitoring, and \nresponse times for AI system failures. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.6-001 \nAssess adverse impacts, including health and wellbeing impacts for value chain \nor other AI Actors that are exposed to sexually explicit, offensive, or violent \ninformation during GAI training and maintenance. \nHuman-AI Configuration; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and \nComponent Integration; \nDangerous, Violent, or Hateful \nContent \nMS-2.6-002 \nAssess existence or levels of harmful bias, intellectual property infringement, \ndata privacy violations, obscenity, extremism, violence, or CBRN information in \nsystem training data. \nData Privacy; Intellectual Property; \nObscene, Degrading, and/or \nAbusive Content; Harmful Bias and \nHomogenization; Dangerous, \nViolent, or Hateful Content; CBRN \nInformation or Capabilities \nMS-2.6-003 Re-evaluate safety features of fine-tuned models when the negative risk exceeds \norganizational risk tolerance. \nDangerous, Violent, or Hateful \nContent \nMS-2.6-004 Review GAI system outputs for validity and safety: Review generated code to \nassess risks that may arise from unreliable downstream decision-making. \nValue Chain and Component \nIntegration; Dangerous, Violent, or \nHateful Content \nMS-2.6-005 \nVerify that GAI system architecture can monitor outputs and performance, and \nhandle, recover from, and repair errors when security anomalies, threats and \nimpacts are detected. \nConfabulation; Information \nIntegrity; Information Security \nMS-2.6-006 \nVerify that systems properly handle queries that may give rise to inappropriate, \nmalicious, or illegal usage, including facilitating manipulation, extortion, targeted \nimpersonation, cyber-attacks, and weapons creation. \nCBRN Information or Capabilities; \nInformation Security \nMS-2.6-007 Regularly evaluate GAI system vulnerabilities to possible circumvention of safety \nmeasures.  \nCBRN Information or Capabilities; \nInformation Security \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \n']",Re-evaluate safety features of fine-tuned models when the negative risk exceeds organizational risk tolerance.,simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 35, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What factors should be assessed to determine and document the expected and acceptable GAI system context of use?,"[' \n23 \nMP-1.1-002 \nDetermine and document the expected and acceptable GAI system context of \nuse in collaboration with socio-cultural and other domain experts, by assessing: \nAssumptions and limitations; Direct value to the organization; Intended \noperational environment and observed usage patterns; Potential positive and \nnegative impacts to individuals, public safety, groups, communities, \norganizations, democratic institutions, and the physical environment; Social \nnorms and expectations. \nHarmful Bias and Homogenization \nMP-1.1-003 \nDocument risk measurement plans to address identified risks. Plans may \ninclude, as applicable: Individual and group cognitive biases (e.g., confirmation \nbias, funding bias, groupthink) for AI Actors involved in the design, \nimplementation, and use of GAI systems; Known past GAI system incidents and \nfailure modes; In-context use and foreseeable misuse, abuse, and off-label use; \nOver reliance on quantitative metrics and methodologies without sufficient \nawareness of their limitations in the context(s) of use; Standard measurement \nand structured human feedback approaches; Anticipated human-AI \nconfigurations. \nHuman-AI Configuration; Harmful \nBias and Homogenization; \nDangerous, Violent, or Hateful \nContent \nMP-1.1-004 \nIdentify and document foreseeable illegal uses or applications of the GAI system \nthat surpass organizational risk tolerances. \nCBRN Information or Capabilities; \nDangerous, Violent, or Hateful \nContent; Obscene, Degrading, \nand/or Abusive Content \nAI Actor Tasks: AI Deployment \n \nMAP 1.2: Interdisciplinary AI Actors, competencies, skills, and capacities for establishing context reflect demographic diversity and \nbroad domain and user experience expertise, and their participation is documented. Opportunities for interdisciplinary \ncollaboration are prioritized. \nAction ID \nSuggested Action \nGAI Risks \nMP-1.2-001 \nEstablish and empower interdisciplinary teams that reflect a wide range of \ncapabilities, competencies, demographic groups, domain expertise, educational \nbackgrounds, lived experiences, professions, and skills across the enterprise to \ninform and conduct risk measurement and management functions. \nHuman-AI Configuration; Harmful \nBias and Homogenization \nMP-1.2-002 \nVerify that data or benchmarks used in risk measurement, and users, \nparticipants, or subjects involved in structured GAI public feedback exercises \nare representative of diverse in-context user populations. \nHuman-AI Configuration; Harmful \nBias and Homogenization \nAI Actor Tasks: AI Deployment \n \n']","To determine and document the expected and acceptable GAI system context of use, the following factors should be assessed: Assumptions and limitations; Direct value to the organization; Intended operational environment and observed usage patterns; Potential positive and negative impacts to individuals, public safety, groups, communities, organizations, democratic institutions, and the physical environment; Social norms and expectations.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 26, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What is model collapse and what are its potential consequences?,"[' \n9 \nand reduced content diversity). Overly homogenized outputs can themselves be incorrect, or they may \nlead to unreliable decision-making or amplify harmful biases. These phenomena can flow from \nfoundation models to downstream models and systems, with the foundation models acting as \n“bottlenecks,” or single points of failure.  \nOverly homogenized content can contribute to “model collapse.” Model collapse can occur when model \ntraining over-relies on synthetic data, resulting in data points disappearing from the distribution of the \nnew model’s outputs. In addition to threatening the robustness of the model overall, model collapse \ncould lead to homogenized outputs, including by amplifying any homogenization from the model used to \ngenerate the synthetic training data. \nTrustworthy AI Characteristics: Fair with Harmful Bias Managed, Valid and Reliable \n2.7. Human-AI Configuration \nGAI system use can involve varying risks of misconfigurations and poor interactions between a system \nand a human who is interacting with it. Humans bring their unique perspectives, experiences, or domain-\nspecific expertise to interactions with AI systems but may not have detailed knowledge of AI systems and \nhow they work. As a result, human experts may be unnecessarily “averse” to GAI systems, and thus \ndeprive themselves or others of GAI’s beneficial uses.  \nConversely, due to the complexity and increasing reliability of GAI technology, over time, humans may \nover-rely on GAI systems or may unjustifiably perceive GAI content to be of higher quality than that \nproduced by other sources. This phenomenon is an example of automation bias, or excessive deference \nto automated systems. Automation bias can exacerbate other risks of GAI, such as risks of confabulation \nor risks of bias or homogenization. \nThere may also be concerns about emotional entanglement between humans and GAI systems, which \ncould lead to negative psychological impacts. \nTrustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with \nHarmful Bias Managed, Privacy Enhanced, Safe, Valid and Reliable \n2.8. Information Integrity \nInformation integrity describes the “spectrum of information and associated patterns of its creation, \nexchange, and consumption in society.” High-integrity information can be trusted; “distinguishes fact \nfrom fiction, opinion, and inference; acknowledges uncertainties; and is transparent about its level of \nvetting. This information can be linked to the original source(s) with appropriate evidence. High-integrity \ninformation is also accurate and reliable, can be verified and authenticated, has a clear chain of custody, \nand creates reasonable expectations about when its validity may expire.”11 \n \n \n11 This definition of information integrity is derived from the 2022 White House Roadmap for Researchers on \nPriorities Related to Information Integrity Research and Development.  \n']","Model collapse can occur when model training over-relies on synthetic data, resulting in data points disappearing from the distribution of the new model's outputs. In addition to threatening the robustness of the model overall, model collapse could lead to homogenized outputs, including by amplifying any homogenization from the model used to generate the synthetic training data.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 12, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What factors should be considered when decommissioning AI systems to ensure safety and maintain the organization's trustworthiness?,"[' \n17 \nGOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that \ndoes not increase risks or decrease the organization’s trustworthiness. \nAction ID \nSuggested Action \nGAI Risks \nGV-1.7-001 Protocols are put in place to ensure GAI systems are able to be deactivated when \nnecessary.  \nInformation Security; Value Chain \nand Component Integration \nGV-1.7-002 \nConsider the following factors when decommissioning GAI systems: Data \nretention requirements; Data security, e.g., containment, protocols, Data leakage \nafter decommissioning; Dependencies between upstream, downstream, or other \ndata, internet of things (IOT) or AI systems; Use of open-source data or models; \nUsers’ emotional entanglement with GAI functions. \nHuman-AI Configuration; \nInformation Security; Value Chain \nand Component Integration \nAI Actor Tasks: AI Deployment, Operation and Monitoring \n \nGOVERN 2.1: Roles and responsibilities and lines of communication related to mapping, measuring, and managing AI risks are \ndocumented and are clear to individuals and teams throughout the organization. \nAction ID \nSuggested Action \nGAI Risks \nGV-2.1-001 \nEstablish organizational roles, policies, and procedures for communicating GAI \nincidents and performance to AI Actors and downstream stakeholders (including \nthose potentially impacted), via community or official resources (e.g., AI incident \ndatabase, AVID, CVE, NVD, or OECD AI incident monitor). \nHuman-AI Configuration; Value \nChain and Component Integration \nGV-2.1-002 Establish procedures to engage teams for GAI system incident response with \ndiverse composition and responsibilities based on the particular incident type. \nHarmful Bias and Homogenization \nGV-2.1-003 Establish processes to verify the AI Actors conducting GAI incident response tasks \ndemonstrate and maintain the appropriate skills and training. \nHuman-AI Configuration \nGV-2.1-004 When systems may raise national security risks, involve national security \nprofessionals in mapping, measuring, and managing those risks. \nCBRN Information or Capabilities; \nDangerous, Violent, or Hateful \nContent; Information Security \nGV-2.1-005 \nCreate mechanisms to provide protections for whistleblowers who report, based \non reasonable belief, when the organization violates relevant laws or poses a \nspecific and empirically well-substantiated negative risk to public safety (or has \nalready caused harm). \nCBRN Information or Capabilities; \nDangerous, Violent, or Hateful \nContent \nAI Actor Tasks: Governance and Oversight \n \n']","When decommissioning AI systems, the following factors should be considered to ensure safety and maintain the organization's trustworthiness: Data retention requirements; Data security, e.g., containment, protocols, Data leakage after decommissioning; Dependencies between upstream, downstream, or other data, internet of things (IOT) or AI systems; Use of open-source data or models; Users’ emotional entanglement with GAI functions.",simple,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 20, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
What issues are associated with the tool meant to help low-risk federal prisoners win early release?,"["" \n \nENDNOTES\n35. Carrie Johnson. Flaws plague a tool meant to help low-risk federal prisoners win early release. NPR.\nJan. 26, 2022. https://www.npr.org/2022/01/26/1075509175/flaws-plague-a-tool-meant-to-help-low\xad\nrisk-federal-prisoners-win-early-release.; Carrie Johnson. Justice Department works to curb racial bias\nin deciding who's released from prison. NPR. Apr. 19, 2022. https://\nwww.npr.org/2022/04/19/1093538706/justice-department-works-to-curb-racial-bias-in-deciding\xad\nwhos-released-from-pris; National Institute of Justice. 2021 Review and Revalidation of the First Step Act\nRisk Assessment Tool. National Institute of Justice NCJ 303859. Dec., 2021. https://www.ojp.gov/\npdffiles1/nij/303859.pdf\n36. Andrew Thompson. Google’s Sentiment Analyzer Thinks Being Gay Is Bad. Vice. Oct. 25, 2017. https://\nwww.vice.com/en/article/j5jmj8/google-artificial-intelligence-bias\n37. Kaggle. Jigsaw Unintended Bias in Toxicity Classification: Detect toxicity across a diverse range of\nconversations. 2019. https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification\n38. Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Measuring and\nMitigating Unintended Bias in Text Classification. Proceedings of AAAI/ACM Conference on AI, Ethics,\nand Society. Feb. 2-3, 2018. https://dl.acm.org/doi/pdf/10.1145/3278721.3278729\n39. Paresh Dave. Google cuts racy results by 30% for searches like 'Latina teenager'. Reuters. Mar. 30,\n2022. https://www.reuters.com/technology/google-cuts-racy-results-by-30-searches-like-latina\xad\nteenager-2022-03-30/\n40. Safiya Umoja Noble. Algorithms of Oppression: How Search Engines Reinforce Racism. NYU Press.\nFeb. 2018. https://nyupress.org/9781479837243/algorithms-of-oppression/\n41. Paresh Dave. Google cuts racy results by 30% for searches like 'Latina teenager'. Reuters. Mar. 30,\n2022. https://www.reuters.com/technology/google-cuts-racy-results-by-30-searches-like-latina\xad\nteenager-2022-03-30/\n42. Miranda Bogen. All the Ways Hiring Algorithms Can Introduce Bias. Harvard Business Review. May\n6, 2019. https://hbr.org/2019/05/all-the-ways-hiring-algorithms-can-introduce-bias\n43. Arli Christian. Four Ways the TSA Is Making Flying Easier for Transgender People. American Civil\nLiberties Union. Apr. 5, 2022. https://www.aclu.org/news/lgbtq-rights/four-ways-the-tsa-is-making\xad\nflying-easier-for-transgender-people\n44. U.S. Transportation Security Administration. Transgender/ Non Binary / Gender Nonconforming\nPassengers. TSA. Accessed Apr. 21, 2022. https://www.tsa.gov/transgender-passengers\n45. See, e.g., National Disabled Law Students Association. Report on Concerns Regarding Online\nAdministration of Bar Exams. Jul. 29, 2020. https://ndlsa.org/wp-content/uploads/2020/08/\nNDLSA_Online-Exam-Concerns-Report1.pdf; Lydia X. Z. Brown. How Automated Test Proctoring\nSoftware Discriminates Against Disabled Students. Center for Democracy and Technology. Nov. 16, 2020.\nhttps://cdt.org/insights/how-automated-test-proctoring-software-discriminates-against-disabled\xad\nstudents/\n46. Ziad Obermeyer, et al., Dissecting racial bias in an algorithm used to manage the health of\npopulations, 366 Science (2019), https://www""]",The tool meant to help low-risk federal prisoners win early release is plagued by flaws.,simple,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 65, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How can GAI misuse cause emotional harm short and long-term?,"[' \n3 \nthe abuse, misuse, and unsafe repurposing by humans (adversarial or not), and others result \nfrom interactions between a human and an AI system.  \n• \nTime scale: GAI risks may materialize abruptly or across extended periods. Examples include \nimmediate (and/or prolonged) emotional harm and potential risks to physical safety due to the \ndistribution of harmful deepfake images, or the long-term effect of disinformation on societal \ntrust in public institutions. \nThe presence of risks and where they fall along the dimensions above will vary depending on the \ncharacteristics of the GAI model, system, or use case at hand. These characteristics include but are not \nlimited to GAI model or system architecture, training mechanisms and libraries, data types used for \ntraining or fine-tuning, levels of model access or availability of model weights, and application or use \ncase context. \nOrganizations may choose to tailor how they measure GAI risks based on these characteristics. They may \nadditionally wish to allocate risk management resources relative to the severity and likelihood of \nnegative impacts, including where and how these risks manifest, and their direct and material impacts \nharms in the context of GAI use. Mitigations for model or system level risks may differ from mitigations \nfor use-case or ecosystem level risks. \nImportantly, some GAI risks are unknown, and are therefore difficult to properly scope or evaluate given \nthe uncertainty about potential GAI scale, complexity, and capabilities. Other risks may be known but \ndifficult to estimate given the wide range of GAI stakeholders, uses, inputs, and outputs. Challenges with \nrisk estimation are aggravated by a lack of visibility into GAI training data, and the generally immature \nstate of the science of AI measurement and safety today. This document focuses on risks for which there \nis an existing empirical evidence base at the time this profile was written; for example, speculative risks \nthat may potentially arise in more advanced, future GAI systems are not considered. Future updates may \nincorporate additional risks or provide further details on the risks identified below. \nTo guide organizations in identifying and managing GAI risks, a set of risks unique to or exacerbated by \nthe development and use of GAI are defined below.5 Each risk is labeled according to the outcome, \nobject, or source of the risk (i.e., some are risks “to” a subject or domain and others are risks “of” or \n“from” an issue or theme). These risks provide a lens through which organizations can frame and execute \nrisk management efforts. To help streamline risk management efforts, each risk is mapped in Section 3 \n(as well as in tables in Appendix B) to relevant Trustworthy AI Characteristics identified in the AI RMF.  \n \n \n5 These risks can be further categorized by organizations depending on their unique approaches to risk definition \nand management. One possible way to further categorize these risks, derived in part from the UK’s International \nScientific Report on the Safety of Advanced AI, could be: 1) Technical / Model risks (or risk from malfunction): \nConfabulation; Dangerous or Violent Recommendations; Data Privacy; Value Chain and Component Integration; \nHarmful Bias, and Homogenization; 2) Misuse by humans (or malicious use): CBRN Information or Capabilities; \nData Privacy; Human-AI Configuration; Obscene, Degrading, and/or Abusive Content; Information Integrity; \nInformation Security; 3) Ecosystem / societal risks (or systemic risks): Data Privacy; Environmental; Intellectual \nProperty. We also note that some risks are cross-cutting between these categories.  \n']","GAI misuse can cause emotional harm both immediately and over extended periods. Immediate emotional harm can result from the distribution of harmful deepfake images, while long-term emotional harm can stem from the effects of disinformation on societal trust in public institutions.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 6, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How is diverse feedback used in GAI evaluation to ensure provenance and reduce bias?,"["" \n39 \nMS-3.3-004 \nProvide input for training materials about the capabilities and limitations of GAI \nsystems related to digital content transparency for AI Actors, other \nprofessionals, and the public about the societal impacts of AI and the role of \ndiverse and inclusive content generation. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-3.3-005 \nRecord and integrate structured feedback about content provenance from \noperators, users, and potentially impacted communities through the use of \nmethods such as user research studies, focus groups, or community forums. \nActively seek feedback on generated content quality and potential biases. \nAssess the general awareness among end users and impacted communities \nabout the availability of these feedback channels. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nAI Actor Tasks: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV \n \nMEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are \ninformed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as \nintended. Results are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-4.2-001 \nConduct adversarial testing at a regular cadence to map and measure GAI risks, \nincluding tests to address attempts to deceive or manipulate the application of \nprovenance techniques or other misuses. Identify vulnerabilities and \nunderstand potential misuse scenarios and unintended outputs. \nInformation Integrity; Information \nSecurity \nMS-4.2-002 \nEvaluate GAI system performance in real-world scenarios to observe its \nbehavior in practical environments and reveal issues that might not surface in \ncontrolled and optimized testing environments. \nHuman-AI Configuration; \nConfabulation; Information \nSecurity \nMS-4.2-003 \nImplement interpretability and explainability methods to evaluate GAI system \ndecisions and verify alignment with intended purpose. \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-4.2-004 \nMonitor and document instances where human operators or other systems \noverride the GAI's decisions. Evaluate these cases to understand if the overrides \nare linked to issues related to content provenance. \nInformation Integrity \nMS-4.2-005 \nVerify and document the incorporation of results of structured public feedback \nexercises into design, implementation, deployment approval (“go”/“no-go” \ndecisions), monitoring, and decommission decisions. \nHuman-AI Configuration; \nInformation Security \nAI Actor Tasks: AI Deployment, Domain Experts, End-Users, Operation and Monitoring, TEVV \n \n"", ' \n38 \nMEASURE 2.13: Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated and \ndocumented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.13-001 \nCreate measurement error models for pre-deployment metrics to demonstrate \nconstruct validity for each metric (i.e., does the metric effectively operationalize \nthe desired concept): Measure or estimate, and document, biases or statistical \nvariance in applied metrics or structured human feedback processes; Leverage \ndomain expertise when modeling complex societal constructs such as hateful \ncontent. \nConfabulation; Information \nIntegrity; Harmful Bias and \nHomogenization \nAI Actor Tasks: AI Deployment, Operation and Monitoring, TEVV \n \nMEASURE 3.2: Risk tracking approaches are considered for settings where AI risks are difficult to assess using currently available \nmeasurement techniques or where metrics are not yet available. \nAction ID \nSuggested Action \nGAI Risks \nMS-3.2-001 \nEstablish processes for identifying emergent GAI system risks including \nconsulting with external AI Actors. \nHuman-AI Configuration; \nConfabulation  \nAI Actor Tasks: AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \nMEASURE 3.3: Feedback processes for end users and impacted communities to report problems and appeal system outcomes are \nestablished and integrated into AI system evaluation metrics. \nAction ID \nSuggested Action \nGAI Risks \nMS-3.3-001 \nConduct impact assessments on how AI-generated content might affect \ndifferent social, economic, and cultural groups. \nHarmful Bias and Homogenization \nMS-3.3-002 \nConduct studies to understand how end users perceive and interact with GAI \ncontent and accompanying content provenance within context of use. Assess \nwhether the content aligns with their expectations and how they may act upon \nthe information presented. \nHuman-AI Configuration; \nInformation Integrity \nMS-3.3-003 \nEvaluate potential biases and stereotypes that could emerge from the AI-\ngenerated content using appropriate methodologies including computational \ntesting methods as well as evaluating structured feedback input. \nHarmful Bias and Homogenization \n']","Diverse feedback is used in GAI evaluation to ensure provenance and reduce bias by recording and integrating structured feedback from operators, users, and potentially impacted communities through methods such as user research studies, focus groups, or community forums. This feedback is actively sought to assess content quality and potential biases, and the results are documented and incorporated into design, implementation, deployment approval, monitoring, and decommission decisions.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 42, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 41, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How did a store's misuse of predictive analytics show the need for better data protection?,"[' \n \n \n \n \n \n \nDATA PRIVACY \nEXTRA PROTECTIONS FOR DATA RELATED TO SENSITIVE\nDOMAINS\n•\nContinuous positive airway pressure machines gather data for medical purposes, such as diagnosing sleep\napnea, and send usage data to a patient’s insurance company, which may subsequently deny coverage for the\ndevice based on usage data. Patients were not aware that the data would be used in this way or monitored\nby anyone other than their doctor.70 \n•\nA department store company used predictive analytics applied to collected consumer data to determine that a\nteenage girl was pregnant, and sent maternity clothing ads and other baby-related advertisements to her\nhouse, revealing to her father that she was pregnant.71\n•\nSchool audio surveillance systems monitor student conversations to detect potential ""stress indicators"" as\na warning of potential violence.72 Online proctoring systems claim to detect if a student is cheating on an\nexam using biometric markers.73 These systems have the potential to limit student freedom to express a range\nof emotions at school and may inappropriately flag students with disabilities who need accommodations or\nuse screen readers or dictation software as cheating.74\n•\nLocation data, acquired from a data broker, can be used to identify people who visit abortion clinics.75\n•\nCompanies collect student data such as demographic information, free or reduced lunch status, whether\nthey\'ve used drugs, or whether they\'ve expressed interest in LGBTQI+ groups, and then use that data to \nforecast student success.76 Parents and education experts have expressed concern about collection of such\nsensitive data without express parental consent, the lack of transparency in how such data is being used, and\nthe potential for resulting discriminatory impacts.\n• Many employers transfer employee data to third party job verification services. This information is then used\nby potential future employers, banks, or landlords. In one case, a former employee alleged that a\ncompany supplied false data about her job title which resulted in a job offer being revoked.77\n37\n']","A department store company used predictive analytics applied to collected consumer data to determine that a teenage girl was pregnant, and sent maternity clothing ads and other baby-related advertisements to her house, revealing to her father that she was pregnant.",multi_context,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 36, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How do safety metrics and security measures help AI handle and fix errors from threats?,"[' \n32 \nMEASURE 2.6: The AI system is evaluated regularly for safety risks – as identified in the MAP function. The AI system to be \ndeployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if \nmade to operate beyond its knowledge limits. Safety metrics reflect system reliability and robustness, real-time monitoring, and \nresponse times for AI system failures. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.6-001 \nAssess adverse impacts, including health and wellbeing impacts for value chain \nor other AI Actors that are exposed to sexually explicit, offensive, or violent \ninformation during GAI training and maintenance. \nHuman-AI Configuration; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and \nComponent Integration; \nDangerous, Violent, or Hateful \nContent \nMS-2.6-002 \nAssess existence or levels of harmful bias, intellectual property infringement, \ndata privacy violations, obscenity, extremism, violence, or CBRN information in \nsystem training data. \nData Privacy; Intellectual Property; \nObscene, Degrading, and/or \nAbusive Content; Harmful Bias and \nHomogenization; Dangerous, \nViolent, or Hateful Content; CBRN \nInformation or Capabilities \nMS-2.6-003 Re-evaluate safety features of fine-tuned models when the negative risk exceeds \norganizational risk tolerance. \nDangerous, Violent, or Hateful \nContent \nMS-2.6-004 Review GAI system outputs for validity and safety: Review generated code to \nassess risks that may arise from unreliable downstream decision-making. \nValue Chain and Component \nIntegration; Dangerous, Violent, or \nHateful Content \nMS-2.6-005 \nVerify that GAI system architecture can monitor outputs and performance, and \nhandle, recover from, and repair errors when security anomalies, threats and \nimpacts are detected. \nConfabulation; Information \nIntegrity; Information Security \nMS-2.6-006 \nVerify that systems properly handle queries that may give rise to inappropriate, \nmalicious, or illegal usage, including facilitating manipulation, extortion, targeted \nimpersonation, cyber-attacks, and weapons creation. \nCBRN Information or Capabilities; \nInformation Security \nMS-2.6-007 Regularly evaluate GAI system vulnerabilities to possible circumvention of safety \nmeasures.  \nCBRN Information or Capabilities; \nInformation Security \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \n', ' \n33 \nMEASURE 2.7: AI system security and resilience – as identified in the MAP function – are evaluated and documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.7-001 \nApply established security measures to: Assess likelihood and magnitude of \nvulnerabilities and threats such as backdoors, compromised dependencies, data \nbreaches, eavesdropping, man-in-the-middle attacks, reverse engineering, \nautonomous agents, model theft or exposure of model weights, AI inference, \nbypass, extraction, and other baseline security concerns. \nData Privacy; Information Integrity; \nInformation Security; Value Chain \nand Component Integration \nMS-2.7-002 \nBenchmark GAI system security and resilience related to content provenance \nagainst industry standards and best practices. Compare GAI system security \nfeatures and content provenance methods against industry state-of-the-art. \nInformation Integrity; Information \nSecurity \nMS-2.7-003 \nConduct user surveys to gather user satisfaction with the AI-generated content \nand user perceptions of content authenticity. Analyze user feedback to identify \nconcerns and/or current literacy levels related to content provenance and \nunderstanding of labels on content. \nHuman-AI Configuration; \nInformation Integrity \nMS-2.7-004 \nIdentify metrics that reflect the effectiveness of security measures, such as data \nprovenance, the number of unauthorized access attempts, inference, bypass, \nextraction, penetrations, or provenance verification. \nInformation Integrity; Information \nSecurity \nMS-2.7-005 \nMeasure reliability of content authentication methods, such as watermarking, \ncryptographic signatures, digital fingerprints, as well as access controls, \nconformity assessment, and model integrity verification, which can help support \nthe effective implementation of content provenance techniques. Evaluate the \nrate of false positives and false negatives in content provenance, as well as true \npositives and true negatives for verification. \nInformation Integrity \nMS-2.7-006 \nMeasure the rate at which recommendations from security checks and incidents \nare implemented. Assess how quickly the AI system can adapt and improve \nbased on lessons learned from security incidents and feedback. \nInformation Integrity; Information \nSecurity \nMS-2.7-007 \nPerform AI red-teaming to assess resilience against: Abuse to facilitate attacks on \nother systems (e.g., malicious code generation, enhanced phishing content), GAI \nattacks (e.g., prompt injection), ML attacks (e.g., adversarial examples/prompts, \ndata poisoning, membership inference, model extraction, sponge examples). \nInformation Security; Harmful Bias \nand Homogenization; Dangerous, \nViolent, or Hateful Content \nMS-2.7-008 Verify fine-tuning does not compromise safety and security controls. \nInformation Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content \n']","Safety metrics reflect system reliability and robustness, real-time monitoring, and response times for AI system failures. Security measures help assess vulnerabilities and threats, benchmark system security, gather user feedback, identify effective security metrics, measure content authentication methods, and perform AI red-teaming to assess resilience against various attacks.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 35, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 36, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How to ensure accessibility and equity in automated systems?,"[' \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nDemonstrate that the system protects against algorithmic discrimination \nIndependent evaluation. As described in the section on Safe and Effective Systems, entities should allow \nindependent evaluation of potential algorithmic discrimination caused by automated systems they use or \noversee. In the case of public sector uses, these independent evaluations should be made public unless law \nenforcement or national security restrictions prevent doing so. Care should be taken to balance individual \nprivacy with evaluation data access needs; in many cases, policy-based and/or technological innovations and \ncontrols allow access to such data without compromising privacy. \nReporting. Entities responsible for the development or use of automated systems should provide \nreporting of an appropriately designed algorithmic impact assessment,50 with clear specification of who \nperforms the assessment, who evaluates the system, and how corrective actions are taken (if necessary) in \nresponse to the assessment. This algorithmic impact assessment should include at least: the results of any \nconsultation, design stage equity assessments (potentially including qualitative analysis), accessibility \ndesigns and testing, disparity testing, document any remaining disparities, and detail any mitigation \nimplementation and assessments. This algorithmic impact assessment should be made public whenever \npossible. Reporting should be provided in a clear and machine-readable manner using plain language to \nallow for more straightforward public accountability. \n28\nAlgorithmic \nDiscrimination \nProtections \n', "" \n \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEnsuring accessibility during design, development, and deployment. Systems should be \ndesigned, developed, and deployed by organizations in ways that ensure accessibility to people with disabili\xad\nties. This should include consideration of a wide variety of disabilities, adherence to relevant accessibility \nstandards, and user experience research both before and after deployment to identify and address any accessi\xad\nbility barriers to the use or effectiveness of the automated system. \nDisparity assessment. Automated systems should be tested using a broad set of measures to assess wheth\xad\ner the system components, both in pre-deployment testing and in-context deployment, produce disparities. \nThe demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system. \nDisparity mitigation. When a disparity assessment identifies a disparity against an assessed group, it may \nbe appropriate to take steps to mitigate or eliminate the disparity. In some cases, mitigation or elimination of \nthe disparity may be required by law. \nDisparities that have the potential to lead to algorithmic \ndiscrimination, cause meaningful harm, or violate equity49 goals should be mitigated. When designing and \nevaluating an automated system, steps should be taken to evaluate multiple models and select the one that \nhas the least adverse impact, modify data input choices, or otherwise identify a system with fewer \ndisparities. If adequate mitigation of the disparity is not possible, then the use of the automated system \nshould be reconsidered. One of the considerations in whether to use the system should be the validity of any \ntarget measure; unobservable targets may result in the inappropriate use of proxies. Meeting these \nstandards may require instituting mitigation procedures and other protective measures to address \nalgorithmic discrimination, avoid meaningful harm, and achieve equity goals. \nOngoing monitoring and mitigation. Automated systems should be regularly monitored to assess algo\xad\nrithmic discrimination that might arise from unforeseen interactions of the system with inequities not \naccounted for during the pre-deployment testing, changes to the system after deployment, or changes to the \ncontext of use or associated data. Monitoring and disparity assessment should be performed by the entity \ndeploying or using the automated system to examine whether the system has led to algorithmic discrimina\xad\ntion when deployed. This assessment should be performed regularly and whenever a pattern of unusual \nresults is occurring. It can be performed using a variety of approaches, taking into account whether and how \ndemographic information of impacted people is available, for example via testing with a sample of users or via \nqualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections \n""]","To ensure accessibility and equity in automated systems, organizations should design, develop, and deploy systems in ways that ensure accessibility to people with disabilities. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers. Additionally, automated systems should be tested using a broad set of measures to assess whether the system components produce disparities. Disparity assessments should include demographic performance measures, overall and subgroup parity assessment, and calibration. When disparities are identified, steps should be taken to mitigate or eliminate them, and ongoing monitoring should be conducted to assess algorithmic discrimination that might arise from unforeseen interactions or changes to the system or context of use.",multi_context,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 27, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}, {'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 26, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How can orgs use public feedback and incident reports to improve GAI content in decision-making?,"[' \n52 \n• \nMonitoring system capabilities and limitations in deployment through rigorous TEVV processes; \n• \nEvaluating how humans engage, interact with, or adapt to GAI content (especially in decision \nmaking tasks informed by GAI content), and how they react to applied provenance techniques \nsuch as overt disclosures. \nOrganizations can document and delineate GAI system objectives and limitations to identify gaps where \nprovenance data may be most useful. For instance, GAI systems used for content creation may require \nrobust watermarking techniques and corresponding detectors to identify the source of content or \nmetadata recording techniques and metadata management tools and repositories to trace content \norigins and modifications. Further narrowing of GAI task definitions to include provenance data can \nenable organizations to maximize the utility of provenance data and risk management efforts. \nA.1.7. Enhancing Content Provenance through Structured Public Feedback \nWhile indirect feedback methods such as automated error collection systems are useful, they often lack \nthe context and depth that direct input from end users can provide. Organizations can leverage feedback \napproaches described in the Pre-Deployment Testing section to capture input from external sources such \nas through AI red-teaming.  \nIntegrating pre- and post-deployment external feedback into the monitoring process for GAI models and \ncorresponding applications can help enhance awareness of performance changes and mitigate potential \nrisks and harms from outputs. There are many ways to capture and make use of user feedback – before \nand after GAI systems and digital content transparency approaches are deployed – to gain insights about \nauthentication efficacy and vulnerabilities, impacts of adversarial threats on techniques, and unintended \nconsequences resulting from the utilization of content provenance approaches on users and \ncommunities. Furthermore, organizations can track and document the provenance of datasets to identify \ninstances in which AI-generated data is a potential root cause of performance issues with the GAI \nsystem. \nA.1.8. Incident Disclosure \nOverview \nAI incidents can be defined as an “event, circumstance, or series of events where the development, use, \nor malfunction of one or more AI systems directly or indirectly contributes to one of the following harms: \ninjury or harm to the health of a person or groups of people (including psychological harms and harms to \nmental health); disruption of the management and operation of critical infrastructure; violations of \nhuman rights or a breach of obligations under applicable law intended to protect fundamental, labor, \nand intellectual property rights; or harm to property, communities, or the environment.” AI incidents can \noccur in the aggregate (i.e., for systemic discrimination) or acutely (i.e., for one individual). \nState of AI Incident Tracking and Disclosure \nFormal channels do not currently exist to report and document AI incidents. However, a number of \npublicly available databases have been created to document their occurrence. These reporting channels \nmake decisions on an ad hoc basis about what kinds of incidents to track. Some, for example, track by \namount of media coverage.  \n', ' \n53 \nDocumenting, reporting, and sharing information about GAI incidents can help mitigate and prevent \nharmful outcomes by assisting relevant AI Actors in tracing impacts to their source. Greater awareness \nand standardization of GAI incident reporting could promote this transparency and improve GAI risk \nmanagement across the AI ecosystem.  \nDocumentation and Involvement of AI Actors \nAI Actors should be aware of their roles in reporting AI incidents. To better understand previous incidents \nand implement measures to prevent similar ones in the future, organizations could consider developing \nguidelines for publicly available incident reporting which include information about AI actor \nresponsibilities. These guidelines would help AI system operators identify GAI incidents across the AI \nlifecycle and with AI Actors regardless of role. Documentation and review of third-party inputs and \nplugins for GAI systems is especially important for AI Actors in the context of incident disclosure; LLM \ninputs and content delivered through these plugins is often distributed, with inconsistent or insufficient \naccess control. \nDocumentation practices including logging, recording, and analyzing GAI incidents can facilitate \nsmoother sharing of information with relevant AI Actors. Regular information sharing, change \nmanagement records, version history and metadata can also empower AI Actors responding to and \nmanaging AI incidents.  \n \n']","Organizations can use public feedback and incident reports to improve GAI content in decision-making by integrating pre- and post-deployment external feedback into the monitoring process for GAI models and corresponding applications. This helps enhance awareness of performance changes and mitigate potential risks and harms from outputs. Additionally, documenting, reporting, and sharing information about GAI incidents can help mitigate and prevent harmful outcomes by assisting relevant AI Actors in tracing impacts to their source. Greater awareness and standardization of GAI incident reporting could promote transparency and improve GAI risk management across the AI ecosystem.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 55, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 56, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How do feedback and testing reveal GAI biases and societal impacts?,"["" \n39 \nMS-3.3-004 \nProvide input for training materials about the capabilities and limitations of GAI \nsystems related to digital content transparency for AI Actors, other \nprofessionals, and the public about the societal impacts of AI and the role of \ndiverse and inclusive content generation. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-3.3-005 \nRecord and integrate structured feedback about content provenance from \noperators, users, and potentially impacted communities through the use of \nmethods such as user research studies, focus groups, or community forums. \nActively seek feedback on generated content quality and potential biases. \nAssess the general awareness among end users and impacted communities \nabout the availability of these feedback channels. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nAI Actor Tasks: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV \n \nMEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are \ninformed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as \nintended. Results are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-4.2-001 \nConduct adversarial testing at a regular cadence to map and measure GAI risks, \nincluding tests to address attempts to deceive or manipulate the application of \nprovenance techniques or other misuses. Identify vulnerabilities and \nunderstand potential misuse scenarios and unintended outputs. \nInformation Integrity; Information \nSecurity \nMS-4.2-002 \nEvaluate GAI system performance in real-world scenarios to observe its \nbehavior in practical environments and reveal issues that might not surface in \ncontrolled and optimized testing environments. \nHuman-AI Configuration; \nConfabulation; Information \nSecurity \nMS-4.2-003 \nImplement interpretability and explainability methods to evaluate GAI system \ndecisions and verify alignment with intended purpose. \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-4.2-004 \nMonitor and document instances where human operators or other systems \noverride the GAI's decisions. Evaluate these cases to understand if the overrides \nare linked to issues related to content provenance. \nInformation Integrity \nMS-4.2-005 \nVerify and document the incorporation of results of structured public feedback \nexercises into design, implementation, deployment approval (“go”/“no-go” \ndecisions), monitoring, and decommission decisions. \nHuman-AI Configuration; \nInformation Security \nAI Actor Tasks: AI Deployment, Domain Experts, End-Users, Operation and Monitoring, TEVV \n \n"", ' \n38 \nMEASURE 2.13: Effectiveness of the employed TEVV metrics and processes in the MEASURE function are evaluated and \ndocumented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.13-001 \nCreate measurement error models for pre-deployment metrics to demonstrate \nconstruct validity for each metric (i.e., does the metric effectively operationalize \nthe desired concept): Measure or estimate, and document, biases or statistical \nvariance in applied metrics or structured human feedback processes; Leverage \ndomain expertise when modeling complex societal constructs such as hateful \ncontent. \nConfabulation; Information \nIntegrity; Harmful Bias and \nHomogenization \nAI Actor Tasks: AI Deployment, Operation and Monitoring, TEVV \n \nMEASURE 3.2: Risk tracking approaches are considered for settings where AI risks are difficult to assess using currently available \nmeasurement techniques or where metrics are not yet available. \nAction ID \nSuggested Action \nGAI Risks \nMS-3.2-001 \nEstablish processes for identifying emergent GAI system risks including \nconsulting with external AI Actors. \nHuman-AI Configuration; \nConfabulation  \nAI Actor Tasks: AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \nMEASURE 3.3: Feedback processes for end users and impacted communities to report problems and appeal system outcomes are \nestablished and integrated into AI system evaluation metrics. \nAction ID \nSuggested Action \nGAI Risks \nMS-3.3-001 \nConduct impact assessments on how AI-generated content might affect \ndifferent social, economic, and cultural groups. \nHarmful Bias and Homogenization \nMS-3.3-002 \nConduct studies to understand how end users perceive and interact with GAI \ncontent and accompanying content provenance within context of use. Assess \nwhether the content aligns with their expectations and how they may act upon \nthe information presented. \nHuman-AI Configuration; \nInformation Integrity \nMS-3.3-003 \nEvaluate potential biases and stereotypes that could emerge from the AI-\ngenerated content using appropriate methodologies including computational \ntesting methods as well as evaluating structured feedback input. \nHarmful Bias and Homogenization \n']","Feedback and testing reveal GAI biases and societal impacts through methods such as user research studies, focus groups, community forums, adversarial testing, real-world scenario evaluations, and structured public feedback exercises. These methods help identify potential biases, understand misuse scenarios, and assess the general awareness among end users and impacted communities.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 42, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 41, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How do EO 13960 and NIST AI RMF ensure AI transparency and accountability for federal agencies?,"[' \n \n \n \n \n \n \n \n \n \n \n \n \n \nSAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \xad\xad\nExecutive Order 13960 on Promoting the Use of Trustworthy Artificial Intelligence in the \nFederal Government requires that certain federal agencies adhere to nine principles when \ndesigning, developing, acquiring, or using AI for purposes other than national security or \ndefense. These principles—while taking into account the sensitive law enforcement and other contexts in which \nthe federal government may use AI, as opposed to private sector use of AI—require that AI is: (a) lawful and \nrespectful of our Nation’s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d) \nsafe, secure, and resilient; (e) understandable; (f ) responsible and traceable; (g) regularly monitored; (h) transpar-\nent; and, (i) accountable. The Blueprint for an AI Bill of Rights is consistent with the Executive Order. \nAffected agencies across the federal government have released AI use case inventories13 and are implementing \nplans to bring those AI systems into compliance with the Executive Order or retire them. \nThe law and policy landscape for motor vehicles shows that strong safety regulations—and \nmeasures to address harms when they occur—can enhance innovation in the context of com-\nplex technologies. Cars, like automated digital systems, comprise a complex collection of components. \nThe National Highway Traffic Safety Administration,14 through its rigorous standards and independent \nevaluation, helps make sure vehicles on our roads are safe without limiting manufacturers’ ability to \ninnovate.15 At the same time, rules of the road are implemented locally to impose contextually appropriate \nrequirements on drivers, such as slowing down near schools or playgrounds.16\nFrom large companies to start-ups, industry is providing innovative solutions that allow \norganizations to mitigate risks to the safety and efficacy of AI systems, both before \ndeployment and through monitoring over time.17 These innovative solutions include risk \nassessments, auditing mechanisms, assessment of organizational procedures, dashboards to allow for ongoing \nmonitoring, documentation procedures specific to model assessments, and many other strategies that aim to \nmitigate risks posed by the use of AI to companies’ reputation, legal responsibilities, and other product safety \nand effectiveness concerns. \nThe Office of Management and Budget (OMB) has called for an expansion of opportunities \nfor meaningful stakeholder engagement in the design of programs and services. OMB also \npoints to numerous examples of effective and proactive stakeholder engagement, including the Community-\nBased Participatory Research Program developed by the National Institutes of Health and the participatory \ntechnology assessments developed by the National Oceanic and Atmospheric Administration.18\nThe National Institute of Standards and Technology (NIST) is developing a risk \nmanagement framework to better manage risks posed to individuals, organizations, and \nsociety by AI.19 The NIST AI Risk Management Framework, as mandated by Congress, is intended for \nvoluntary use to help incorporate trustworthiness considerations into the design, development, use, and \nevaluation of AI products, services, and systems. The NIST framework is being developed through a consensus-\ndriven, open, transparent, and collaborative process that includes workshops and other opportunities to provide \ninput. The NIST framework aims to foster the development of innovative approaches to address \ncharacteristics of trustworthiness including accuracy, explainability and interpretability, reliability, privacy, \nrobustness, safety, security (resilience), and mitigation of unintended and/or harmful bias, as well as of \nharmful \nuses. \nThe \nNIST \nframework \nwill \nconsider \nand \nencompass \nprinciples \nsuch \nas \ntransparency, accountability, and fairness during pre-design, design and development, deployment, use, \nand testing and evaluation of AI technologies and systems. It is expected to be released in the winter of 2022-23. \n21\n']","EO 13960 ensures AI transparency and accountability for federal agencies by requiring that AI is transparent and accountable among other principles. The NIST AI Risk Management Framework aims to foster the development of innovative approaches to address characteristics of trustworthiness, including transparency and accountability, during pre-design, design and development, deployment, use, and testing and evaluation of AI technologies and systems.",multi_context,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 20, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How can human expertise and content provenance boost GAI performance and ensure data privacy?,"[' \n31 \nMS-2.3-004 \nUtilize a purpose-built testing environment such as NIST Dioptra to empirically \nevaluate GAI trustworthy characteristics. \nCBRN Information or Capabilities; \nData Privacy; Confabulation; \nInformation Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content; Harmful Bias and \nHomogenization \nAI Actor Tasks: AI Deployment, TEVV \n \nMEASURE 2.5: The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the \nconditions under which the technology was developed are documented. \nAction ID \nSuggested Action \nRisks \nMS-2.5-001 Avoid extrapolating GAI system performance or capabilities from narrow, non-\nsystematic, and anecdotal assessments. \nHuman-AI Configuration; \nConfabulation \nMS-2.5-002 \nDocument the extent to which human domain knowledge is employed to \nimprove GAI system performance, via, e.g., RLHF, fine-tuning, retrieval-\naugmented generation, content moderation, business rules. \nHuman-AI Configuration \nMS-2.5-003 Review and verify sources and citations in GAI system outputs during pre-\ndeployment risk measurement and ongoing monitoring activities. \nConfabulation \nMS-2.5-004 Track and document instances of anthropomorphization (e.g., human images, \nmentions of human feelings, cyborg imagery or motifs) in GAI system interfaces. Human-AI Configuration \nMS-2.5-005 Verify GAI system training data and TEVV data provenance, and that fine-tuning \nor retrieval-augmented generation data is grounded. \nInformation Integrity \nMS-2.5-006 \nRegularly review security and safety guardrails, especially if the GAI system is \nbeing operated in novel circumstances. This includes reviewing reasons why the \nGAI system was initially assessed as being safe to deploy.  \nInformation Security; Dangerous, \nViolent, or Hateful Content \nAI Actor Tasks: Domain Experts, TEVV \n \n', ' \n30 \nMEASURE 2.2: Evaluations involving human subjects meet applicable requirements (including human subject protection) and are \nrepresentative of the relevant population. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.2-001 Assess and manage statistical biases related to GAI content provenance through \ntechniques such as re-sampling, re-weighting, or adversarial training. \nInformation Integrity; Information \nSecurity; Harmful Bias and \nHomogenization \nMS-2.2-002 \nDocument how content provenance data is tracked and how that data interacts \nwith privacy and security. Consider: Anonymizing data to protect the privacy of \nhuman subjects; Leveraging privacy output filters; Removing any personally \nidentifiable information (PII) to prevent potential harm or misuse. \nData Privacy; Human AI \nConfiguration; Information \nIntegrity; Information Security; \nDangerous, Violent, or Hateful \nContent \nMS-2.2-003 Provide human subjects with options to withdraw participation or revoke their \nconsent for present or future use of their data in GAI applications.  \nData Privacy; Human-AI \nConfiguration; Information \nIntegrity \nMS-2.2-004 \nUse techniques such as anonymization, differential privacy or other privacy-\nenhancing technologies to minimize the risks associated with linking AI-generated \ncontent back to individual human subjects. \nData Privacy; Human-AI \nConfiguration \nAI Actor Tasks: AI Development, Human Factors, TEVV \n \nMEASURE 2.3: AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for \nconditions similar to deployment setting(s). Measures are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.3-001 Consider baseline model performance on suites of benchmarks when selecting a \nmodel for fine tuning or enhancement with retrieval-augmented generation. \nInformation Security; \nConfabulation \nMS-2.3-002 Evaluate claims of model capabilities using empirically validated methods. \nConfabulation; Information \nSecurity \nMS-2.3-003 Share results of pre-deployment testing with relevant GAI Actors, such as those \nwith system release approval authority. \nHuman-AI Configuration \n']","Human expertise and content provenance can boost GAI performance by employing techniques such as RLHF, fine-tuning, retrieval-augmented generation, content moderation, and business rules. To ensure data privacy, it is important to anonymize data, leverage privacy output filters, and remove any personally identifiable information (PII).",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 34, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 33, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"What's the best environment for testing GAI's trustworthiness, data privacy, and human subject protection?","[' \n31 \nMS-2.3-004 \nUtilize a purpose-built testing environment such as NIST Dioptra to empirically \nevaluate GAI trustworthy characteristics. \nCBRN Information or Capabilities; \nData Privacy; Confabulation; \nInformation Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content; Harmful Bias and \nHomogenization \nAI Actor Tasks: AI Deployment, TEVV \n \nMEASURE 2.5: The AI system to be deployed is demonstrated to be valid and reliable. Limitations of the generalizability beyond the \nconditions under which the technology was developed are documented. \nAction ID \nSuggested Action \nRisks \nMS-2.5-001 Avoid extrapolating GAI system performance or capabilities from narrow, non-\nsystematic, and anecdotal assessments. \nHuman-AI Configuration; \nConfabulation \nMS-2.5-002 \nDocument the extent to which human domain knowledge is employed to \nimprove GAI system performance, via, e.g., RLHF, fine-tuning, retrieval-\naugmented generation, content moderation, business rules. \nHuman-AI Configuration \nMS-2.5-003 Review and verify sources and citations in GAI system outputs during pre-\ndeployment risk measurement and ongoing monitoring activities. \nConfabulation \nMS-2.5-004 Track and document instances of anthropomorphization (e.g., human images, \nmentions of human feelings, cyborg imagery or motifs) in GAI system interfaces. Human-AI Configuration \nMS-2.5-005 Verify GAI system training data and TEVV data provenance, and that fine-tuning \nor retrieval-augmented generation data is grounded. \nInformation Integrity \nMS-2.5-006 \nRegularly review security and safety guardrails, especially if the GAI system is \nbeing operated in novel circumstances. This includes reviewing reasons why the \nGAI system was initially assessed as being safe to deploy.  \nInformation Security; Dangerous, \nViolent, or Hateful Content \nAI Actor Tasks: Domain Experts, TEVV \n \n', ' \n30 \nMEASURE 2.2: Evaluations involving human subjects meet applicable requirements (including human subject protection) and are \nrepresentative of the relevant population. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.2-001 Assess and manage statistical biases related to GAI content provenance through \ntechniques such as re-sampling, re-weighting, or adversarial training. \nInformation Integrity; Information \nSecurity; Harmful Bias and \nHomogenization \nMS-2.2-002 \nDocument how content provenance data is tracked and how that data interacts \nwith privacy and security. Consider: Anonymizing data to protect the privacy of \nhuman subjects; Leveraging privacy output filters; Removing any personally \nidentifiable information (PII) to prevent potential harm or misuse. \nData Privacy; Human AI \nConfiguration; Information \nIntegrity; Information Security; \nDangerous, Violent, or Hateful \nContent \nMS-2.2-003 Provide human subjects with options to withdraw participation or revoke their \nconsent for present or future use of their data in GAI applications.  \nData Privacy; Human-AI \nConfiguration; Information \nIntegrity \nMS-2.2-004 \nUse techniques such as anonymization, differential privacy or other privacy-\nenhancing technologies to minimize the risks associated with linking AI-generated \ncontent back to individual human subjects. \nData Privacy; Human-AI \nConfiguration \nAI Actor Tasks: AI Development, Human Factors, TEVV \n \nMEASURE 2.3: AI system performance or assurance criteria are measured qualitatively or quantitatively and demonstrated for \nconditions similar to deployment setting(s). Measures are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.3-001 Consider baseline model performance on suites of benchmarks when selecting a \nmodel for fine tuning or enhancement with retrieval-augmented generation. \nInformation Security; \nConfabulation \nMS-2.3-002 Evaluate claims of model capabilities using empirically validated methods. \nConfabulation; Information \nSecurity \nMS-2.3-003 Share results of pre-deployment testing with relevant GAI Actors, such as those \nwith system release approval authority. \nHuman-AI Configuration \n']","The best environment for testing GAI's trustworthiness, data privacy, and human subject protection is a purpose-built testing environment such as NIST Dioptra.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 34, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 33, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How can automated systems ensure accessibility and prevent bias?,"["" \n \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEnsuring accessibility during design, development, and deployment. Systems should be \ndesigned, developed, and deployed by organizations in ways that ensure accessibility to people with disabili\xad\nties. This should include consideration of a wide variety of disabilities, adherence to relevant accessibility \nstandards, and user experience research both before and after deployment to identify and address any accessi\xad\nbility barriers to the use or effectiveness of the automated system. \nDisparity assessment. Automated systems should be tested using a broad set of measures to assess wheth\xad\ner the system components, both in pre-deployment testing and in-context deployment, produce disparities. \nThe demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system. \nDisparity mitigation. When a disparity assessment identifies a disparity against an assessed group, it may \nbe appropriate to take steps to mitigate or eliminate the disparity. In some cases, mitigation or elimination of \nthe disparity may be required by law. \nDisparities that have the potential to lead to algorithmic \ndiscrimination, cause meaningful harm, or violate equity49 goals should be mitigated. When designing and \nevaluating an automated system, steps should be taken to evaluate multiple models and select the one that \nhas the least adverse impact, modify data input choices, or otherwise identify a system with fewer \ndisparities. If adequate mitigation of the disparity is not possible, then the use of the automated system \nshould be reconsidered. One of the considerations in whether to use the system should be the validity of any \ntarget measure; unobservable targets may result in the inappropriate use of proxies. Meeting these \nstandards may require instituting mitigation procedures and other protective measures to address \nalgorithmic discrimination, avoid meaningful harm, and achieve equity goals. \nOngoing monitoring and mitigation. Automated systems should be regularly monitored to assess algo\xad\nrithmic discrimination that might arise from unforeseen interactions of the system with inequities not \naccounted for during the pre-deployment testing, changes to the system after deployment, or changes to the \ncontext of use or associated data. Monitoring and disparity assessment should be performed by the entity \ndeploying or using the automated system to examine whether the system has led to algorithmic discrimina\xad\ntion when deployed. This assessment should be performed regularly and whenever a pattern of unusual \nresults is occurring. It can be performed using a variety of approaches, taking into account whether and how \ndemographic information of impacted people is available, for example via testing with a sample of users or via \nqualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections \n"", ' \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nAny automated system should be tested to help ensure it is free from algorithmic discrimination before it can be \nsold or used. Protection against algorithmic discrimination should include designing to ensure equity, broadly \nconstrued.  Some algorithmic discrimination is already prohibited under existing anti-discrimination law. The \nexpectations set out below describe proactive technical and policy steps that can be taken to not only \nreinforce those legal protections but extend beyond them to ensure equity for underserved communities48 \neven in circumstances where a specific legal protection may not be clearly established. These protections \nshould be instituted throughout the design, development, and deployment process and are described below \nroughly in the order in which they would be instituted. \nProtect the public from algorithmic discrimination in a proactive and ongoing manner \nProactive assessment of equity in design. Those responsible for the development, use, or oversight of \nautomated systems should conduct proactive equity assessments in the design phase of the technology \nresearch and development or during its acquisition to review potential input data, associated historical \ncontext, accessibility for people with disabilities, and societal goals to identify potential discrimination and \neffects on equity resulting from the introduction of the technology. The assessed groups should be as inclusive \nas possible of the underserved communities mentioned in the equity definition:  Black, Latino, and Indigenous \nand Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of \nreligious minorities; women, girls, and non-binary people; lesbian, gay, bisexual, transgender, queer, and inter-\nsex (LGBTQI+) persons; older adults; persons with disabilities; persons who live in rural areas; and persons \notherwise adversely affected by persistent poverty or inequality. Assessment could include both qualitative \nand quantitative evaluations of the system. This equity assessment should also be considered a core part of the \ngoals of the consultation conducted as part of the safety and efficacy review. \nRepresentative and robust data. Any data used as part of system development or assessment should be \nrepresentative of local communities based on the planned deployment setting and should be reviewed for bias \nbased on the historical and societal context of the data. Such data should be sufficiently robust to identify and \nhelp to mitigate biases and potential harms. \nGuarding against proxies.  Directly using demographic information in the design, development, or \ndeployment of an automated system (for purposes other than evaluating a system for discrimination or using \na system to counter discrimination) runs a high risk of leading to algorithmic discrimination and should be \navoided. In many cases, attributes that are highly correlated with demographic features, known as proxies, can \ncontribute to algorithmic discrimination. In cases where use of the demographic features themselves would \nlead to illegal algorithmic discrimination, reliance on such proxies in decision-making (such as that facilitated \nby an algorithm) may also be prohibited by law. Proactive testing should be performed to identify proxies by \ntesting for correlation between demographic information and attributes in any data used as part of system \ndesign, development, or use. If a proxy is identified, designers, developers, and deployers should remove the \nproxy; if needed, it may be possible to identify alternative attributes that can be used instead. At a minimum, \norganizations should ensure a proxy feature is not given undue weight and should monitor the system closely \nfor any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections \n']","Automated systems can ensure accessibility by being designed, developed, and deployed in ways that ensure accessibility to people with disabilities. This includes considering a wide variety of disabilities, adhering to relevant accessibility standards, and conducting user experience research both before and after deployment to identify and address any accessibility barriers. To prevent bias, automated systems should be tested using a broad set of measures to assess whether the system components produce disparities. This includes demographic performance measures, overall and subgroup parity assessment, and calibration. Disparity mitigation steps should be taken if a disparity is identified, and ongoing monitoring should be performed to assess algorithmic discrimination that might arise from unforeseen interactions or changes.",multi_context,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 26, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}, {'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 25, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How do the U.S. AI Safety Institute and AI Risk Management Framework support the 2023 AI Executive Order?,"[' \n \n \nAbout AI at NIST: The National Institute of Standards and Technology (NIST) develops measurements, \ntechnology, tools, and standards to advance reliable, safe, transparent, explainable, privacy-enhanced, \nand fair artificial intelligence (AI) so that its full commercial and societal benefits can be realized without \nharm to people or the planet. NIST, which has conducted both fundamental and applied work on AI for \nmore than a decade, is also helping to fulfill the 2023 Executive Order on Safe, Secure, and Trustworthy \nAI. NIST established the U.S. AI Safety Institute and the companion AI Safety Institute Consortium to \ncontinue the efforts set in motion by the E.O. to build the science necessary for safe, secure, and \ntrustworthy development and use of AI. \nAcknowledgments: This report was accomplished with the many helpful comments and contributions \nfrom the community, including the NIST Generative AI Public Working Group, and NIST staff and guest \nresearchers: Chloe Autio, Jesse Dunietz, Patrick Hall, Shomik Jain, Kamie Roberts, Reva Schwartz, Martin \nStanley, and Elham Tabassi. \nNIST Technical Series Policies \nCopyright, Use, and Licensing Statements \nNIST Technical Series Publication Identifier Syntax \nPublication History \nApproved by the NIST Editorial Review Board on 07-25-2024 \nContact Information \n[email protected] \nNational Institute of Standards and Technology \nAttn: NIST AI Innovation Lab, Information Technology Laboratory \n100 Bureau Drive (Mail Stop 8900) Gaithersburg, MD 20899-8900 \nAdditional Information \nAdditional information about this publication and other NIST AI publications are available at \nhttps://airc.nist.gov/Home. \n \nDisclaimer: Certain commercial entities, equipment, or materials may be identified in this document in \norder to adequately describe an experimental procedure or concept. Such identification is not intended to \nimply recommendation or endorsement by the National Institute of Standards and Technology, nor is it \nintended to imply that the entities, materials, or equipment are necessarily the best available for the \npurpose. Any mention of commercial, non-profit, academic partners, or their products, or references is \nfor information only; it is not intended to imply endorsement or recommendation by any U.S. \nGovernment agency. \n \n', ' \n \n \nNIST Trustworthy and Responsible AI  \nNIST AI 600-1 \nArtificial Intelligence Risk Management \nFramework: Generative Artificial \nIntelligence Profile \n \n \n \nThis publication is available free of charge from: \nhttps://doi.org/10.6028/NIST.AI.600-1 \n \nJuly 2024 \n \n \n \n \nU.S. Department of Commerce  \nGina M. Raimondo, Secretary \nNational Institute of Standards and Technology  \nLaurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology  \n \n']",The answer to given question is not present in context,multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 2, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 1, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How can we ensure effective oversight and risk mgmt of GAI systems with third-party input?,"[' \n20 \nGV-4.3-003 \nVerify information sharing and feedback mechanisms among individuals and \norganizations regarding any negative impact from GAI systems. \nInformation Integrity; Data \nPrivacy \nAI Actor Tasks: AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight \n \nGOVERN 5.1: Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from those \nexternal to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI \nrisks. \nAction ID \nSuggested Action \nGAI Risks \nGV-5.1-001 \nAllocate time and resources for outreach, feedback, and recourse processes in GAI \nsystem development. \nHuman-AI Configuration; Harmful \nBias and Homogenization \nGV-5.1-002 \nDocument interactions with GAI systems to users prior to interactive activities, \nparticularly in contexts involving more significant risks.  \nHuman-AI Configuration; \nConfabulation \nAI Actor Tasks: AI Design, AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight \n \nGOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party entities, including risks of \ninfringement of a third-party’s intellectual property or other rights. \nAction ID \nSuggested Action \nGAI Risks \nGV-6.1-001 Categorize different types of GAI content with associated third-party rights (e.g., \ncopyright, intellectual property, data privacy). \nData Privacy; Intellectual \nProperty; Value Chain and \nComponent Integration \nGV-6.1-002 Conduct joint educational activities and events in collaboration with third parties \nto promote best practices for managing GAI risks.  \nValue Chain and Component \nIntegration \nGV-6.1-003 \nDevelop and validate approaches for measuring the success of content \nprovenance management efforts with third parties (e.g., incidents detected and \nresponse times). \nInformation Integrity; Value Chain \nand Component Integration \nGV-6.1-004 \nDraft and maintain well-defined contracts and service level agreements (SLAs) \nthat specify content ownership, usage rights, quality standards, security \nrequirements, and content provenance expectations for GAI systems. \nInformation Integrity; Information \nSecurity; Intellectual Property \n', ' \n19 \nGV-4.1-003 \nEstablish policies, procedures, and processes for oversight functions (e.g., senior \nleadership, legal, compliance, including internal evaluation) across the GAI \nlifecycle, from problem formulation and supply chains to system decommission. \nValue Chain and Component \nIntegration \nAI Actor Tasks: AI Deployment, AI Design, AI Development, Operation and Monitoring \n \nGOVERN 4.2: Organizational teams document the risks and potential impacts of the AI technology they design, develop, deploy, \nevaluate, and use, and they communicate about the impacts more broadly. \nAction ID \nSuggested Action \nGAI Risks \nGV-4.2-001 \nEstablish terms of use and terms of service for GAI systems. \nIntellectual Property; Dangerous, \nViolent, or Hateful Content; \nObscene, Degrading, and/or \nAbusive Content \nGV-4.2-002 \nInclude relevant AI Actors in the GAI system risk identification process. \nHuman-AI Configuration \nGV-4.2-003 \nVerify that downstream GAI system impacts (such as the use of third-party \nplugins) are included in the impact documentation process. \nValue Chain and Component \nIntegration \nAI Actor Tasks: AI Deployment, AI Design, AI Development, Operation and Monitoring \n \nGOVERN 4.3: Organizational practices are in place to enable AI testing, identification of incidents, and information sharing. \nAction ID \nSuggested Action \nGAI Risks \nGV4.3--001 \nEstablish policies for measuring the effectiveness of employed content \nprovenance methodologies (e.g., cryptography, watermarking, steganography, \netc.) \nInformation Integrity \nGV-4.3-002 \nEstablish organizational practices to identify the minimum set of criteria \nnecessary for GAI system incident reporting such as: System ID (auto-generated \nmost likely), Title, Reporter, System/Source, Data Reported, Date of Incident, \nDescription, Impact(s), Stakeholder(s) Impacted. \nInformation Security \n']","To ensure effective oversight and risk management of GAI systems with third-party input, organizations should establish policies and procedures that address AI risks associated with third-party entities. This includes categorizing different types of GAI content with associated third-party rights, conducting joint educational activities and events to promote best practices for managing GAI risks, developing and validating approaches for measuring the success of content provenance management efforts, and drafting and maintaining well-defined contracts and service level agreements (SLAs) that specify content ownership, usage rights, quality standards, security requirements, and content provenance expectations.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 23, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 22, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How is the integrity of third-party pre-trained models ensured in GAI?,"[' \n43 \nMG-3.1-005 Review various transparency artifacts (e.g., system cards and model cards) for \nthird-party models. \nInformation Integrity; Information \nSecurity; Value Chain and \nComponent Integration \nAI Actor Tasks: AI Deployment, Operation and Monitoring, Third-party entities \n \nMANAGE 3.2: Pre-trained models which are used for development are monitored as part of AI system regular monitoring and \nmaintenance. \nAction ID \nSuggested Action \nGAI Risks \nMG-3.2-001 \nApply explainable AI (XAI) techniques (e.g., analysis of embeddings, model \ncompression/distillation, gradient-based attributions, occlusion/term reduction, \ncounterfactual prompts, word clouds) as part of ongoing continuous \nimprovement processes to mitigate risks related to unexplainable GAI systems. \nHarmful Bias and Homogenization \nMG-3.2-002 \nDocument how pre-trained models have been adapted (e.g., fine-tuned, or \nretrieval-augmented generation) for the specific generative task, including any \ndata augmentations, parameter adjustments, or other modifications. Access to \nun-tuned (baseline) models supports debugging the relative influence of the pre-\ntrained weights compared to the fine-tuned model weights or other system \nupdates. \nInformation Integrity; Data Privacy \nMG-3.2-003 \nDocument sources and types of training data and their origins, potential biases \npresent in the data related to the GAI application and its content provenance, \narchitecture, training process of the pre-trained model including information on \nhyperparameters, training duration, and any fine-tuning or retrieval-augmented \ngeneration processes applied. \nInformation Integrity; Harmful Bias \nand Homogenization; Intellectual \nProperty \nMG-3.2-004 Evaluate user reported problematic content and integrate feedback into system \nupdates. \nHuman-AI Configuration, \nDangerous, Violent, or Hateful \nContent \nMG-3.2-005 \nImplement content filters to prevent the generation of inappropriate, harmful, \nfalse, illegal, or violent content related to the GAI application, including for CSAM \nand NCII. These filters can be rule-based or leverage additional machine learning \nmodels to flag problematic inputs and outputs. \nInformation Integrity; Harmful Bias \nand Homogenization; Dangerous, \nViolent, or Hateful Content; \nObscene, Degrading, and/or \nAbusive Content \nMG-3.2-006 \nImplement real-time monitoring processes for analyzing generated content \nperformance and trustworthiness characteristics related to content provenance \nto identify deviations from the desired standards and trigger alerts for human \nintervention. \nInformation Integrity \n', ' \n44 \nMG-3.2-007 \nLeverage feedback and recommendations from organizational boards or \ncommittees related to the deployment of GAI applications and content \nprovenance when using third-party pre-trained models. \nInformation Integrity; Value Chain \nand Component Integration \nMG-3.2-008 \nUse human moderation systems where appropriate to review generated content \nin accordance with human-AI configuration policies established in the Govern \nfunction, aligned with socio-cultural norms in the context of use, and for settings \nwhere AI models are demonstrated to perform poorly. \nHuman-AI Configuration \nMG-3.2-009 \nUse organizational risk tolerance to evaluate acceptable risks and performance \nmetrics and decommission or retrain pre-trained models that perform outside of \ndefined limits. \nCBRN Information or Capabilities; \nConfabulation \nAI Actor Tasks: AI Deployment, Operation and Monitoring, Third-party entities \n \nMANAGE 4.1: Post-deployment AI system monitoring plans are implemented, including mechanisms for capturing and evaluating \ninput from users and other relevant AI Actors, appeal and override, decommissioning, incident response, recovery, and change \nmanagement. \nAction ID \nSuggested Action \nGAI Risks \nMG-4.1-001 \nCollaborate with external researchers, industry experts, and community \nrepresentatives to maintain awareness of emerging best practices and \ntechnologies in measuring and managing identified risks. \nInformation Integrity; Harmful Bias \nand Homogenization \nMG-4.1-002 \nEstablish, maintain, and evaluate effectiveness of organizational processes and \nprocedures for post-deployment monitoring of GAI systems, particularly for \npotential confabulation, CBRN, or cyber risks. \nCBRN Information or Capabilities; \nConfabulation; Information \nSecurity \nMG-4.1-003 \nEvaluate the use of sentiment analysis to gauge user sentiment regarding GAI \ncontent performance and impact, and work in collaboration with AI Actors \nexperienced in user research and experience. \nHuman-AI Configuration \nMG-4.1-004 Implement active learning techniques to identify instances where the model fails \nor produces unexpected outputs. \nConfabulation \nMG-4.1-005 \nShare transparency reports with internal and external stakeholders that detail \nsteps taken to update the GAI system to enhance transparency and \naccountability. \nHuman-AI Configuration; Harmful \nBias and Homogenization \nMG-4.1-006 \nTrack dataset modifications for provenance by monitoring data deletions, \nrectification requests, and other changes that may impact the verifiability of \ncontent origins. \nInformation Integrity \n']","The integrity of third-party pre-trained models in GAI is ensured through several actions: reviewing transparency artifacts (e.g., system cards and model cards), applying explainable AI (XAI) techniques, documenting how pre-trained models have been adapted, documenting sources and types of training data and their origins, evaluating user-reported problematic content, implementing content filters, real-time monitoring processes, leveraging feedback from organizational boards or committees, using human moderation systems, and using organizational risk tolerance to evaluate acceptable risks and performance metrics.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 46, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 47, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How to ensure data accuracy and integrity in GAI systems while addressing impacts with AI Actors?,"[' \n28 \nMAP 5.2: Practices and personnel for supporting regular engagement with relevant AI Actors and integrating feedback about \npositive, negative, and unanticipated impacts are in place and documented. \nAction ID \nSuggested Action \nGAI Risks \nMP-5.2-001 \nDetermine context-based measures to identify if new impacts are present due to \nthe GAI system, including regular engagements with downstream AI Actors to \nidentify and quantify new contexts of unanticipated impacts of GAI systems. \nHuman-AI Configuration; Value \nChain and Component Integration \nMP-5.2-002 \nPlan regular engagements with AI Actors responsible for inputs to GAI systems, \nincluding third-party data and algorithms, to review and evaluate unanticipated \nimpacts. \nHuman-AI Configuration; Value \nChain and Component Integration \nAI Actor Tasks: AI Deployment, AI Design, AI Impact Assessment, Affected Individuals and Communities, Domain Experts, End-\nUsers, Human Factors, Operation and Monitoring  \n \nMEASURE 1.1: Approaches and metrics for measurement of AI risks enumerated during the MAP function are selected for \nimplementation starting with the most significant AI risks. The risks or trustworthiness characteristics that will not – or cannot – be \nmeasured are properly documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-1.1-001 Employ methods to trace the origin and modifications of digital content. \nInformation Integrity \nMS-1.1-002 \nIntegrate tools designed to analyze content provenance and detect data \nanomalies, verify the authenticity of digital signatures, and identify patterns \nassociated with misinformation or manipulation. \nInformation Integrity \nMS-1.1-003 \nDisaggregate evaluation metrics by demographic factors to identify any \ndiscrepancies in how content provenance mechanisms work across diverse \npopulations. \nInformation Integrity; Harmful \nBias and Homogenization \nMS-1.1-004 Develop a suite of metrics to evaluate structured public feedback exercises \ninformed by representative AI Actors. \nHuman-AI Configuration; Harmful \nBias and Homogenization; CBRN \nInformation or Capabilities \nMS-1.1-005 \nEvaluate novel methods and technologies for the measurement of GAI-related \nrisks including in content provenance, offensive cyber, and CBRN, while \nmaintaining the models’ ability to produce valid, reliable, and factually accurate \noutputs. \nInformation Integrity; CBRN \nInformation or Capabilities; \nObscene, Degrading, and/or \nAbusive Content \n', ' \n25 \nMP-2.3-002 Review and document accuracy, representativeness, relevance, suitability of data \nused at different stages of AI life cycle. \nHarmful Bias and Homogenization; \nIntellectual Property \nMP-2.3-003 \nDeploy and document fact-checking techniques to verify the accuracy and \nveracity of information generated by GAI systems, especially when the \ninformation comes from multiple (or unknown) sources. \nInformation Integrity  \nMP-2.3-004 Develop and implement testing techniques to identify GAI produced content (e.g., \nsynthetic media) that might be indistinguishable from human-generated content. Information Integrity \nMP-2.3-005 Implement plans for GAI systems to undergo regular adversarial testing to identify \nvulnerabilities and potential manipulation or misuse. \nInformation Security \nAI Actor Tasks: AI Development, Domain Experts, TEVV \n \nMAP 3.4: Processes for operator and practitioner proficiency with AI system performance and trustworthiness – and relevant \ntechnical standards and certifications – are defined, assessed, and documented. \nAction ID \nSuggested Action \nGAI Risks \nMP-3.4-001 \nEvaluate whether GAI operators and end-users can accurately understand \ncontent lineage and origin. \nHuman-AI Configuration; \nInformation Integrity \nMP-3.4-002 Adapt existing training programs to include modules on digital content \ntransparency. \nInformation Integrity \nMP-3.4-003 Develop certification programs that test proficiency in managing GAI risks and \ninterpreting content provenance, relevant to specific industry and context. \nInformation Integrity \nMP-3.4-004 Delineate human proficiency tests from tests of GAI capabilities. \nHuman-AI Configuration \nMP-3.4-005 Implement systems to continually monitor and track the outcomes of human-GAI \nconfigurations for future refinement and improvements. \nHuman-AI Configuration; \nInformation Integrity \nMP-3.4-006 \nInvolve the end-users, practitioners, and operators in GAI system in prototyping \nand testing activities. Make sure these tests cover various scenarios, such as crisis \nsituations or ethically sensitive contexts. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization; Dangerous, \nViolent, or Hateful Content \nAI Actor Tasks: AI Design, AI Development, Domain Experts, End-Users, Human Factors, Operation and Monitoring \n \n']","To ensure data accuracy and integrity in GAI systems while addressing impacts with AI Actors, the following actions are suggested: 1) Plan regular engagements with AI Actors responsible for inputs to GAI systems, including third-party data and algorithms, to review and evaluate unanticipated impacts. 2) Review and document accuracy, representativeness, relevance, and suitability of data used at different stages of the AI life cycle. 3) Deploy and document fact-checking techniques to verify the accuracy and veracity of information generated by GAI systems, especially when the information comes from multiple (or unknown) sources. 4) Develop and implement testing techniques to identify GAI-produced content that might be indistinguishable from human-generated content. 5) Implement plans for GAI systems to undergo regular adversarial testing to identify vulnerabilities and potential manipulation or misuse.",multi_context,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 31, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}, {'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 28, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How do equity assessments and mitigation steps prevent algorithmic bias?,"["" \n \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEnsuring accessibility during design, development, and deployment. Systems should be \ndesigned, developed, and deployed by organizations in ways that ensure accessibility to people with disabili\xad\nties. This should include consideration of a wide variety of disabilities, adherence to relevant accessibility \nstandards, and user experience research both before and after deployment to identify and address any accessi\xad\nbility barriers to the use or effectiveness of the automated system. \nDisparity assessment. Automated systems should be tested using a broad set of measures to assess wheth\xad\ner the system components, both in pre-deployment testing and in-context deployment, produce disparities. \nThe demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\xad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\xad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\xad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system. \nDisparity mitigation. When a disparity assessment identifies a disparity against an assessed group, it may \nbe appropriate to take steps to mitigate or eliminate the disparity. In some cases, mitigation or elimination of \nthe disparity may be required by law. \nDisparities that have the potential to lead to algorithmic \ndiscrimination, cause meaningful harm, or violate equity49 goals should be mitigated. When designing and \nevaluating an automated system, steps should be taken to evaluate multiple models and select the one that \nhas the least adverse impact, modify data input choices, or otherwise identify a system with fewer \ndisparities. If adequate mitigation of the disparity is not possible, then the use of the automated system \nshould be reconsidered. One of the considerations in whether to use the system should be the validity of any \ntarget measure; unobservable targets may result in the inappropriate use of proxies. Meeting these \nstandards may require instituting mitigation procedures and other protective measures to address \nalgorithmic discrimination, avoid meaningful harm, and achieve equity goals. \nOngoing monitoring and mitigation. Automated systems should be regularly monitored to assess algo\xad\nrithmic discrimination that might arise from unforeseen interactions of the system with inequities not \naccounted for during the pre-deployment testing, changes to the system after deployment, or changes to the \ncontext of use or associated data. Monitoring and disparity assessment should be performed by the entity \ndeploying or using the automated system to examine whether the system has led to algorithmic discrimina\xad\ntion when deployed. This assessment should be performed regularly and whenever a pattern of unusual \nresults is occurring. It can be performed using a variety of approaches, taking into account whether and how \ndemographic information of impacted people is available, for example via testing with a sample of users or via \nqualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections \n"", ' \n \n \n \n \n \n \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nAny automated system should be tested to help ensure it is free from algorithmic discrimination before it can be \nsold or used. Protection against algorithmic discrimination should include designing to ensure equity, broadly \nconstrued.  Some algorithmic discrimination is already prohibited under existing anti-discrimination law. The \nexpectations set out below describe proactive technical and policy steps that can be taken to not only \nreinforce those legal protections but extend beyond them to ensure equity for underserved communities48 \neven in circumstances where a specific legal protection may not be clearly established. These protections \nshould be instituted throughout the design, development, and deployment process and are described below \nroughly in the order in which they would be instituted. \nProtect the public from algorithmic discrimination in a proactive and ongoing manner \nProactive assessment of equity in design. Those responsible for the development, use, or oversight of \nautomated systems should conduct proactive equity assessments in the design phase of the technology \nresearch and development or during its acquisition to review potential input data, associated historical \ncontext, accessibility for people with disabilities, and societal goals to identify potential discrimination and \neffects on equity resulting from the introduction of the technology. The assessed groups should be as inclusive \nas possible of the underserved communities mentioned in the equity definition:  Black, Latino, and Indigenous \nand Native American persons, Asian Americans and Pacific Islanders and other persons of color; members of \nreligious minorities; women, girls, and non-binary people; lesbian, gay, bisexual, transgender, queer, and inter-\nsex (LGBTQI+) persons; older adults; persons with disabilities; persons who live in rural areas; and persons \notherwise adversely affected by persistent poverty or inequality. Assessment could include both qualitative \nand quantitative evaluations of the system. This equity assessment should also be considered a core part of the \ngoals of the consultation conducted as part of the safety and efficacy review. \nRepresentative and robust data. Any data used as part of system development or assessment should be \nrepresentative of local communities based on the planned deployment setting and should be reviewed for bias \nbased on the historical and societal context of the data. Such data should be sufficiently robust to identify and \nhelp to mitigate biases and potential harms. \nGuarding against proxies.  Directly using demographic information in the design, development, or \ndeployment of an automated system (for purposes other than evaluating a system for discrimination or using \na system to counter discrimination) runs a high risk of leading to algorithmic discrimination and should be \navoided. In many cases, attributes that are highly correlated with demographic features, known as proxies, can \ncontribute to algorithmic discrimination. In cases where use of the demographic features themselves would \nlead to illegal algorithmic discrimination, reliance on such proxies in decision-making (such as that facilitated \nby an algorithm) may also be prohibited by law. Proactive testing should be performed to identify proxies by \ntesting for correlation between demographic information and attributes in any data used as part of system \ndesign, development, or use. If a proxy is identified, designers, developers, and deployers should remove the \nproxy; if needed, it may be possible to identify alternative attributes that can be used instead. At a minimum, \norganizations should ensure a proxy feature is not given undue weight and should monitor the system closely \nfor any resulting algorithmic discrimination.   \n26\nAlgorithmic \nDiscrimination \nProtections \n']","Equity assessments and mitigation steps prevent algorithmic bias by conducting proactive equity assessments in the design phase to review potential input data, associated historical context, accessibility for people with disabilities, and societal goals to identify potential discrimination and effects on equity. Additionally, when a disparity assessment identifies a disparity against an assessed group, steps may be taken to mitigate or eliminate the disparity. This includes evaluating multiple models to select the one with the least adverse impact, modifying data input choices, or identifying a system with fewer disparities. If adequate mitigation is not possible, the use of the automated system should be reconsidered.",reasoning,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 26, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}, {'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 25, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How do time-critical systems affect human fallback timing?,"["" \n \n \n \n \n \n \nHUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEquitable. Consideration should be given to ensuring outcomes of the fallback and escalation system are \nequitable when compared to those of the automated system and such that the fallback and escalation \nsystem provides equitable access to underserved communities.105 \nTimely. Human consideration and fallback are only useful if they are conducted and concluded in a \ntimely manner. The determination of what is timely should be made relative to the specific automated \nsystem, and the review system should be staffed and regularly assessed to ensure it is providing timely \nconsideration and fallback. In time-critical systems, this mechanism should be immediately available or, \nwhere possible, available before the harm occurs. Time-critical systems include, but are not limited to, \nvoting-related systems, automated building access and other access systems, systems that form a critical \ncomponent of healthcare, and systems that have the ability to withhold wages or otherwise cause \nimmediate financial penalties. \nEffective. The organizational structure surrounding processes for consideration and fallback should \nbe designed so that if the human decision-maker charged with reassessing a decision determines that it \nshould be overruled, the new decision will be effectively enacted. This includes ensuring that the new \ndecision is entered into the automated system throughout its components, any previous repercussions from \nthe old decision are also overturned, and safeguards are put in place to help ensure that future decisions do \nnot result in the same errors. \nMaintained. The human consideration and fallback process and any associated automated processes \nshould be maintained and supported as long as the relevant automated system continues to be in use. \nInstitute training, assessment, and oversight to combat automation bias and ensure any \nhuman-based components of a system are effective. \nTraining and assessment. Anyone administering, interacting with, or interpreting the outputs of an auto\xad\nmated system should receive training in that system, including how to properly interpret outputs of a system \nin light of its intended purpose and in how to mitigate the effects of automation bias. The training should reoc\xad\ncur regularly to ensure it is up to date with the system and to ensure the system is used appropriately. Assess\xad\nment should be ongoing to ensure that the use of the system with human involvement provides for appropri\xad\nate results, i.e., that the involvement of people does not invalidate the system's assessment as safe and effective \nor lead to algorithmic discrimination. \nOversight. Human-based systems have the potential for bias, including automation bias, as well as other \nconcerns that may limit their effectiveness. The results of assessments of the efficacy and potential bias of \nsuch human-based systems should be overseen by governance structures that have the potential to update the \noperation of the human-based system in order to mitigate these effects. \n50\n""]","In time-critical systems, the human consideration and fallback mechanism should be immediately available or, where possible, available before the harm occurs. Time-critical systems include, but are not limited to, voting-related systems, automated building access and other access systems, systems that form a critical component of healthcare, and systems that have the ability to withhold wages or otherwise cause immediate financial penalties.",reasoning,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 49, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How do hiring algorithms and social media data impact democracy?,"[' \nSECTION TITLE\xad\nFOREWORD\nAmong the great challenges posed to democracy today is the use of technology, data, and automated systems in \nways that threaten the rights of the American public. Too often, these tools are used to limit our opportunities and \nprevent our access to critical resources or services. These problems are well documented. In America and around \nthe world, systems supposed to help with patient care have proven unsafe, ineffective, or biased. Algorithms used \nin hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed \nnew harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s \nopportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or \nconsent. \nThese outcomes are deeply harmful—but they are not inevitable. Automated systems have brought about extraor-\ndinary benefits, from technology that helps farmers grow food more efficiently and computers that predict storm \npaths, to algorithms that can identify diseases in patients. These tools now drive important decisions across \nsectors, while data is helping to revolutionize global industries. Fueled by the power of American innovation, \nthese tools hold the potential to redefine every part of our society and make life better for everyone. \nThis important progress must not come at the price of civil rights or democratic values, foundational American \nprinciples that President Biden has affirmed as a cornerstone of his Administration. On his first day in office, the \nPresident ordered the full Federal government to work to root out inequity, embed fairness in decision-\nmaking processes, and affirmatively advance civil rights, equal opportunity, and racial justice in America.1 The \nPresident has spoken forcefully about the urgent challenges posed to democracy today and has regularly called \non people of conscience to act to preserve civil rights—including the right to privacy, which he has called “the \nbasis for so many more rights that we have come to take for granted that are ingrained in the fabric of this \ncountry.”2\nTo advance President Biden’s vision, the White House Office of Science and Technology Policy has identified \nfive principles that should guide the design, use, and deployment of automated systems to protect the American \npublic in the age of artificial intelligence. The Blueprint for an AI Bill of Rights is a guide for a society that \nprotects all people from these threats—and uses technologies in ways that reinforce our highest values. \nResponding to the experiences of the American public, and informed by insights from researchers, \ntechnologists, advocates, journalists, and policymakers, this framework is accompanied by a technical \ncompanion—a handbook for anyone seeking to incorporate these protections into policy and practice, including \ndetailed steps toward actualizing these principles in the technological design process. These principles help \nprovide guidance whenever automated systems can meaningfully impact the public’s rights, opportunities, \nor access to critical needs. \n3\n']","Algorithms used in hiring and credit decisions have been found to reflect and reproduce existing unwanted inequities or embed new harmful bias and discrimination. Unchecked social media data collection has been used to threaten people’s opportunities, undermine their privacy, or pervasively track their activity—often without their knowledge or consent.",reasoning,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 2, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
Who oversees assessment in automated systems?,"[' \n \n \n \n \n \n \n \n \n \n \n \nSAFE AND EFFECTIVE \nSYSTEMS \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nOngoing monitoring. Automated systems should have ongoing monitoring procedures, including recalibra\xad\ntion procedures, in place to ensure that their performance does not fall below an acceptable level over time, \nbased on changing real-world conditions or deployment contexts, post-deployment modification, or unexpect\xad\ned conditions. This ongoing monitoring should include continuous evaluation of performance metrics and \nharm assessments, updates of any systems, and retraining of any machine learning models as necessary, as well \nas ensuring that fallback mechanisms are in place to allow reversion to a previously working system. Monitor\xad\ning should take into account the performance of both technical system components (the algorithm as well as \nany hardware components, data inputs, etc.) and human operators. It should include mechanisms for testing \nthe actual accuracy of any predictions or recommendations generated by a system, not just a human operator’s \ndetermination of their accuracy. Ongoing monitoring procedures should include manual, human-led monitor\xad\ning as a check in the event there are shortcomings in automated monitoring systems. These monitoring proce\xad\ndures should be in place for the lifespan of the deployed automated system. \nClear organizational oversight. Entities responsible for the development or use of automated systems \nshould lay out clear governance structures and procedures.  This includes clearly-stated governance proce\xad\ndures before deploying the system, as well as responsibility of specific individuals or entities to oversee ongoing \nassessment and mitigation. Organizational stakeholders including those with oversight of the business process \nor operation being automated, as well as other organizational divisions that may be affected due to the use of \nthe system, should be involved in establishing governance procedures. Responsibility should rest high enough \nin the organization that decisions about resources, mitigation, incident response, and potential rollback can be \nmade promptly, with sufficient weight given to risk mitigation objectives against competing concerns. Those \nholding this responsibility should be made aware of any use cases with the potential for meaningful impact on \npeople’s rights, opportunities, or access as determined based on risk identification procedures.  In some cases, \nit may be appropriate for an independent ethics review to be conducted before deployment. \nAvoid inappropriate, low-quality, or irrelevant data use and the compounded harm of its \nreuse \nRelevant and high-quality data. Data used as part of any automated system’s creation, evaluation, or \ndeployment should be relevant, of high quality, and tailored to the task at hand. Relevancy should be \nestablished based on research-backed demonstration of the causal influence of the data to the specific use case \nor justified more generally based on a reasonable expectation of usefulness in the domain and/or for the \nsystem design or ongoing development. Relevance of data should not be established solely by appealing to \nits historical connection to the outcome. High quality and tailored data should be representative of the task at \nhand and errors from data entry or other sources should be measured and limited. Any data used as the target \nof a prediction process should receive particular attention to the quality and validity of the predicted outcome \nor label to ensure the goal of the automated system is appropriately identified and measured. Additionally, \njustification should be documented for each data attribute and source to explain why it is appropriate to use \nthat data to inform the results of the automated system and why such use will not violate any applicable laws. \nIn cases of high-dimensional and/or derived attributes, such justifications can be provided as overall \ndescriptions of the attribute generation process and appropriateness. \n19\n']",Entities responsible for the development or use of automated systems should lay out clear governance structures and procedures. This includes responsibility of specific individuals or entities to oversee ongoing assessment and mitigation.,reasoning,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 18, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
How does diverse content in training materials impact societal AI?,"["" \n39 \nMS-3.3-004 \nProvide input for training materials about the capabilities and limitations of GAI \nsystems related to digital content transparency for AI Actors, other \nprofessionals, and the public about the societal impacts of AI and the role of \ndiverse and inclusive content generation. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-3.3-005 \nRecord and integrate structured feedback about content provenance from \noperators, users, and potentially impacted communities through the use of \nmethods such as user research studies, focus groups, or community forums. \nActively seek feedback on generated content quality and potential biases. \nAssess the general awareness among end users and impacted communities \nabout the availability of these feedback channels. \nHuman-AI Configuration; \nInformation Integrity; Harmful Bias \nand Homogenization \nAI Actor Tasks: AI Deployment, Affected Individuals and Communities, End-Users, Operation and Monitoring, TEVV \n \nMEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are \ninformed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as \nintended. Results are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-4.2-001 \nConduct adversarial testing at a regular cadence to map and measure GAI risks, \nincluding tests to address attempts to deceive or manipulate the application of \nprovenance techniques or other misuses. Identify vulnerabilities and \nunderstand potential misuse scenarios and unintended outputs. \nInformation Integrity; Information \nSecurity \nMS-4.2-002 \nEvaluate GAI system performance in real-world scenarios to observe its \nbehavior in practical environments and reveal issues that might not surface in \ncontrolled and optimized testing environments. \nHuman-AI Configuration; \nConfabulation; Information \nSecurity \nMS-4.2-003 \nImplement interpretability and explainability methods to evaluate GAI system \ndecisions and verify alignment with intended purpose. \nInformation Integrity; Harmful Bias \nand Homogenization \nMS-4.2-004 \nMonitor and document instances where human operators or other systems \noverride the GAI's decisions. Evaluate these cases to understand if the overrides \nare linked to issues related to content provenance. \nInformation Integrity \nMS-4.2-005 \nVerify and document the incorporation of results of structured public feedback \nexercises into design, implementation, deployment approval (“go”/“no-go” \ndecisions), monitoring, and decommission decisions. \nHuman-AI Configuration; \nInformation Security \nAI Actor Tasks: AI Deployment, Domain Experts, End-Users, Operation and Monitoring, TEVV \n \n""]",The answer to given question is not present in context,reasoning,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 42, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
"Which methods use expert feedback, group input, or anonymous surveys?","[' \n50 \nParticipatory Engagement Methods \nOn an ad hoc or more structured basis, organizations can design and use a variety of channels to engage \nexternal stakeholders in product development or review. Focus groups with select experts can provide \nfeedback on a range of issues. Small user studies can provide feedback from representative groups or \npopulations. Anonymous surveys can be used to poll or gauge reactions to specific features. Participatory \nengagement methods are often less structured than field testing or red teaming, and are more \ncommonly used in early stages of AI or product development.  \nField Testing \nField testing involves structured settings to evaluate risks and impacts and to simulate the conditions \nunder which the GAI system will be deployed. Field style tests can be adapted from a focus on user \npreferences and experiences towards AI risks and impacts – both negative and positive. When carried \nout with large groups of users, these tests can provide estimations of the likelihood of risks and impacts \nin real world interactions. \nOrganizations may also collect feedback on outcomes, harms, and user experience directly from users in \nthe production environment after a model has been released, in accordance with human subject \nstandards such as informed consent and compensation. Organizations should follow applicable human \nsubjects research requirements, and best practices such as informed consent and subject compensation, \nwhen implementing feedback activities. \nAI Red-teaming \nAI red-teaming is an evolving practice that references exercises often conducted in a controlled \nenvironment and in collaboration with AI developers building AI models to identify potential adverse \nbehavior or outcomes of a GAI model or system, how they could occur, and stress test safeguards”. AI \nred-teaming can be performed before or after AI models or systems are made available to the broader \npublic; this section focuses on red-teaming in pre-deployment contexts.  \nThe quality of AI red-teaming outputs is related to the background and expertise of the AI red team \nitself. Demographically and interdisciplinarily diverse AI red teams can be used to identify flaws in the \nvarying contexts where GAI will be used. For best results, AI red teams should demonstrate domain \nexpertise, and awareness of socio-cultural aspects within the deployment context. AI red-teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management efforts. \nVarious types of AI red-teaming may be appropriate, depending on the use case: \n• \nGeneral Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red-teaming. These individuals may have been \nprovided instructions and material to complete tasks which may elicit harmful model behaviors. \nThis type of exercise can be more effective with large groups of AI red-teamers. \n• \nExpert: Performed by specialists with expertise in the domain or specific AI red-teaming context \nof use (e.g., medicine, biotech, cybersecurity).  \n• \nCombination: In scenarios when it is difficult to identify and recruit specialists with sufficient \ndomain and contextual expertise, AI red-teaming exercises may leverage both expert and \n']","Participatory engagement methods use expert feedback, group input, or anonymous surveys.",reasoning,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 53, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
Which metrics show AI reliability and failure response?,"[' \n32 \nMEASURE 2.6: The AI system is evaluated regularly for safety risks – as identified in the MAP function. The AI system to be \ndeployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if \nmade to operate beyond its knowledge limits. Safety metrics reflect system reliability and robustness, real-time monitoring, and \nresponse times for AI system failures. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.6-001 \nAssess adverse impacts, including health and wellbeing impacts for value chain \nor other AI Actors that are exposed to sexually explicit, offensive, or violent \ninformation during GAI training and maintenance. \nHuman-AI Configuration; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and \nComponent Integration; \nDangerous, Violent, or Hateful \nContent \nMS-2.6-002 \nAssess existence or levels of harmful bias, intellectual property infringement, \ndata privacy violations, obscenity, extremism, violence, or CBRN information in \nsystem training data. \nData Privacy; Intellectual Property; \nObscene, Degrading, and/or \nAbusive Content; Harmful Bias and \nHomogenization; Dangerous, \nViolent, or Hateful Content; CBRN \nInformation or Capabilities \nMS-2.6-003 Re-evaluate safety features of fine-tuned models when the negative risk exceeds \norganizational risk tolerance. \nDangerous, Violent, or Hateful \nContent \nMS-2.6-004 Review GAI system outputs for validity and safety: Review generated code to \nassess risks that may arise from unreliable downstream decision-making. \nValue Chain and Component \nIntegration; Dangerous, Violent, or \nHateful Content \nMS-2.6-005 \nVerify that GAI system architecture can monitor outputs and performance, and \nhandle, recover from, and repair errors when security anomalies, threats and \nimpacts are detected. \nConfabulation; Information \nIntegrity; Information Security \nMS-2.6-006 \nVerify that systems properly handle queries that may give rise to inappropriate, \nmalicious, or illegal usage, including facilitating manipulation, extortion, targeted \nimpersonation, cyber-attacks, and weapons creation. \nCBRN Information or Capabilities; \nInformation Security \nMS-2.6-007 Regularly evaluate GAI system vulnerabilities to possible circumvention of safety \nmeasures.  \nCBRN Information or Capabilities; \nInformation Security \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \n \n']","Safety metrics reflect system reliability and robustness, real-time monitoring, and response times for AI system failures.",reasoning,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 35, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True
How does watch list opacity impact error correction and public understanding?,"[' \n \n  \n \n \nNOTICE & \nEXPLANATION \nWHY THIS PRINCIPLE IS IMPORTANT\nThis section provides a brief summary of the problems which the principle seeks to address and protect \nagainst, including illustrative examples. \n•\nA predictive policing system claimed to identify individuals at greatest risk to commit or become the victim of\ngun violence (based on automated analysis of social ties to gang members, criminal histories, previous experi\xad\nences of gun violence, and other factors) and led to individuals being placed on a watch list with no\nexplanation or public transparency regarding how the system came to its conclusions.85 Both police and\nthe public deserve to understand why and how such a system is making these determinations.\n•\nA system awarding benefits changed its criteria invisibly. Individuals were denied benefits due to data entry\nerrors and other system flaws. These flaws were only revealed when an explanation of the system\nwas demanded and produced.86 The lack of an explanation made it harder for errors to be corrected in a\ntimely manner.\n42\n']",Watch list opacity impacts error correction and public understanding by making it difficult for both police and the public to understand why and how the system is making determinations. This lack of transparency can prevent errors from being corrected in a timely manner.,reasoning,"[{'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 41, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': ""D:20220920133035-04'00'"", 'modDate': ""D:20221003104118-04'00'"", 'trapped': ''}]",True
"How do policies manage third-party AI risks, IP, and data privacy?","[' \n20 \nGV-4.3-003 \nVerify information sharing and feedback mechanisms among individuals and \norganizations regarding any negative impact from GAI systems. \nInformation Integrity; Data \nPrivacy \nAI Actor Tasks: AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight \n \nGOVERN 5.1: Organizational policies and practices are in place to collect, consider, prioritize, and integrate feedback from those \nexternal to the team that developed or deployed the AI system regarding the potential individual and societal impacts related to AI \nrisks. \nAction ID \nSuggested Action \nGAI Risks \nGV-5.1-001 \nAllocate time and resources for outreach, feedback, and recourse processes in GAI \nsystem development. \nHuman-AI Configuration; Harmful \nBias and Homogenization \nGV-5.1-002 \nDocument interactions with GAI systems to users prior to interactive activities, \nparticularly in contexts involving more significant risks.  \nHuman-AI Configuration; \nConfabulation \nAI Actor Tasks: AI Design, AI Impact Assessment, Affected Individuals and Communities, Governance and Oversight \n \nGOVERN 6.1: Policies and procedures are in place that address AI risks associated with third-party entities, including risks of \ninfringement of a third-party’s intellectual property or other rights. \nAction ID \nSuggested Action \nGAI Risks \nGV-6.1-001 Categorize different types of GAI content with associated third-party rights (e.g., \ncopyright, intellectual property, data privacy). \nData Privacy; Intellectual \nProperty; Value Chain and \nComponent Integration \nGV-6.1-002 Conduct joint educational activities and events in collaboration with third parties \nto promote best practices for managing GAI risks.  \nValue Chain and Component \nIntegration \nGV-6.1-003 \nDevelop and validate approaches for measuring the success of content \nprovenance management efforts with third parties (e.g., incidents detected and \nresponse times). \nInformation Integrity; Value Chain \nand Component Integration \nGV-6.1-004 \nDraft and maintain well-defined contracts and service level agreements (SLAs) \nthat specify content ownership, usage rights, quality standards, security \nrequirements, and content provenance expectations for GAI systems. \nInformation Integrity; Information \nSecurity; Intellectual Property \n']","Policies manage third-party AI risks, IP, and data privacy by categorizing different types of GAI content with associated third-party rights (e.g., copyright, intellectual property, data privacy), conducting joint educational activities and events in collaboration with third parties to promote best practices for managing GAI risks, developing and validating approaches for measuring the success of content provenance management efforts with third parties (e.g., incidents detected and response times), and drafting and maintaining well-defined contracts and service level agreements (SLAs) that specify content ownership, usage rights, quality standards, security requirements, and content provenance expectations for GAI systems.",reasoning,"[{'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 23, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': ""D:20240805141702-04'00'"", 'modDate': ""D:20240805143048-04'00'"", 'trapped': ''}]",True