Mid term assignment complete pending url and video

.chainlit/config.toml

@@ -0,0 +1,84 @@
+[project]
+# Whether to enable telemetry (default: true). No personal data is collected.
+enable_telemetry = true
+
+# List of environment variables to be provided by each user to use the app.
+user_env = []
+
+# Duration (in seconds) during which the session is saved when the connection is lost
+session_timeout = 3600
+
+# Enable third parties caching (e.g LangChain cache)
+cache = false
+
+# Follow symlink for asset mount (see https://github.com/Chainlit/chainlit/issues/317)
+# follow_symlink = false
+
+[features]
+# Show the prompt playground
+prompt_playground = true
+
+# Process and display HTML in messages. This can be a security risk (see https://stackoverflow.com/questions/19603097/why-is-it-dangerous-to-render-user-generated-html-or-javascript)
+unsafe_allow_html = false
+
+# Process and display mathematical expressions. This can clash with "$" characters in messages.
+latex = false
+
+# Authorize users to upload files with messages
+multi_modal = true
+
+# Allows user to use speech to text
+[features.speech_to_text]
+    enabled = false
+    # See all languages here https://github.com/JamesBrill/react-speech-recognition/blob/HEAD/docs/API.md#language-string
+    # language = "en-US"
+
+[UI]
+# Name of the app and chatbot.
+name = "Chatbot"
+
+# Show the readme while the conversation is empty.
+show_readme_as_default = true
+
+# Description of the app and chatbot. This is used for HTML tags.
+# description = ""
+
+# Large size content are by default collapsed for a cleaner ui
+default_collapse_content = true
+
+# The default value for the expand messages settings.
+default_expand_messages = false
+
+# Hide the chain of thought details from the user in the UI.
+hide_cot = false
+
+# Link to your github repo. This will add a github button in the UI's header.
+# github = ""
+
+# Specify a CSS file that can be used to customize the user interface.
+# The CSS file can be served from the public directory or via an external link.
+# custom_css = "/public/test.css"
+
+# Override default MUI light theme. (Check theme.ts)
+[UI.theme.light]
+    #background = "#FAFAFA"
+    #paper = "#FFFFFF"
+
+    [UI.theme.light.primary]
+        #main = "#F80061"
+        #dark = "#980039"
+        #light = "#FFE7EB"
+
+# Override default MUI dark theme. (Check theme.ts)
+[UI.theme.dark]
+    #background = "#FAFAFA"
+    #paper = "#FFFFFF"
+
+    [UI.theme.dark.primary]
+        #main = "#F80061"
+        #dark = "#980039"
+        #light = "#FFE7EB"
+
+
+[meta]
+generated_by = "0.7.700"

.gitignore

@@ -0,0 +1,9 @@
+venv/
+__pycache__/*
+.env
+download-hf-model.ipynb
+temp
+start_qdrant_services.sh
+requirements copy.txt
+Dockerfile copy
+.venv/

Dockerfile

@@ -0,0 +1,11 @@
+FROM python:3.11.9
+RUN useradd -m -u 1000 user
+USER user
+ENV HOME=/home/user \
+    PATH=/home/user/.local/bin:$PATH
+WORKDIR $HOME/app
+COPY --chown=user . $HOME/app
+COPY ./requirements.txt ~/app/requirements.txt
+RUN pip install -r requirements.txt
+COPY . .
+CMD ["chainlit", "run", "app.py", "--port", "7860"]

README.md

@@ -1,11 +1,73 @@
----
-title: SafeGuardAI
-emoji: 🐠
-colorFrom: yellow
-colorTo: yellow
-sdk: docker
-pinned: false
-license: apache-2.0
----
-
-Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference
+# Midterm
+
+## Background and Context
+The CEO and corporate, with permission of the board, have assembled a crack data science and engineering team to take advantage of RAG, agents, and all of the latest open-source technologies emerging in the industry.  This time it's for real though.  This time, the company is aiming squarely at some Return On Investment - some ROI - on its research and development dollars.
+
+## The Problem
+You are an AI Solutions Engineer.  You've worked directly with internal stakeholders to identify a problem: `people are concerned about the implications of AI, and no one seems to understand the right way to think about building ethical and useful AI applications for enterprises.`
+
+This is a big problem and one that is rapidly changing.  Several people you interviewed said that *they could benefit from a chatbot that helped them understand how the AI industry is evolving, especially as it relates to politics.*  Many are interested due to the current election cycle, but others feel that some of the best guidance is likely to come from the government.
+
+## Task 1: Dealing with the Data
+You identify the following important documents that, if used for context, you believe will help people understand what’s happening now:
+1. 2022: [Blueprint for an AI Bill of Rights: Making Automated Systems Work for the American People](https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf) (PDF)
+2. 2024: [National Institute of Standards and Technology (NIST) Artificial Intelligent Risk Management Framework](https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf) (PDF)
+Your boss, the SVP of Technology, green-lighted this project to drive the adoption of AI throughout the enterprise.  It will be a nice showpiece for the upcoming conference and the big AI initiative announcement the CEO is planning.
+
+> Task 1: Review the two PDFs and decide how best to chunk up the data with a single strategy to optimally answer the variety of questions you expect to receive from people.
+>
+> *Hint: Create a list of potential questions that people are likely to ask!*
+
+✅Deliverables:
+1. Describe the default chunking strategy that you will use.
+2. Articulate a chunking strategy that you would also like to test out.
+3. Describe how and why you made these decisions
+
+## Task 2: Building a Quick End-to-End Prototype
+**You are an AI Systems Engineer.**  The SVP of Technology has tasked you with spinning up a quick RAG prototype for answering questions that internal stakeholders have about AI, using the data provided in Task 1.
+
+> Task 2: Build an end-to-end RAG application using an industry-standard open-source stack and your choice of commercial off-the-shelf models
+
+✅Deliverables:
+1. Build a prototype and deploy to a Hugging Face Space, and create a short (< 2 min) loom video demonstrating some initial testing inputs and outputs.
+2. How did you choose your stack, and why did you select each tool the way you did?
+
+## Task 3: Creating a Golden Test Data Set
+**You are an AI Evaluation & Performance Engineer.**  The AI Systems Engineer who built the initial RAG system has asked for your help and expertise in creating a "Golden Data Set."
+
+> Task 3: Generate a synthetic test data set and baseline an initial evaluation
+
+✅Deliverables:
+1. Assess your pipeline using the RAGAS framework including key metrics faithfulness, answer relevancy, context precision, and context recall.  Provide a table of your output results.
+2. What conclusions can you draw about performance and effectiveness of your pipeline with this information?
+
+## Task 4: Fine-Tuning Open-Source Embeddings
+**You are an Machine Learning Engineer.**  The AI Evaluation and Performance Engineer has asked for your help in fine-tuning the embedding model used in their recent RAG application build.
+
+> Task 4: Generate synthetic fine-tuning data and complete fine-tuning of the open-source embedding model
+
+✅Deliverables:
+1. Swap out your existing embedding model for the new fine-tuned version.  Provide a link to your fine-tuned embedding model on the Hugging Face Hub.
+2. How did you choose the embedding model for this application?
+
+## Task 5: Assessing Performance
+**You are the AI Evaluation & Performance Engineer.**  It's time to assess all options for this product.
+
+> Task 5: Assess the performance of 1) the fine-tuned model, and 2) the two proposed chunking strategies
+
+✅Deliverables:
+1. Test the fine-tuned embedding model using the RAGAS frameworks to quantify any improvements.  Provide results in a table.
+2. Test the two chunking strategies using the RAGAS frameworks to quantify any improvements. Provide results in a table.
+3. The AI Solutions Engineer asks you “Which one is the best to test with internal stakeholders next week, and why?”
+
+## Task 6: Managing Your Boss and User Expectations
+**You are the SVP of Technology.**  Given the work done by your team so far, you're now sitting down with the AI Solutions Engineer.  You have tasked the solutions engineer to test out the new application with at least 50 different internal stakeholders over the next month.
+1. What is the story that you will give to the CEO to tell the whole company at the launch next month?
+2. There appears to be important information not included in our build, for instance, the [270-day update](https://www.whitehouse.gov/briefing-room/statements-releases/2024/07/26/fact-sheet-biden-harris-administration-announces-new-ai-actions-and-receives-additional-major-voluntary-commitment-on-ai/) on the 2023 executive order on [Safe, Secure, and Trustworthy AI](https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/).  How might you incorporate relevant white-house briefing information into future versions?
+
+## Your Final Submission
+Please include the following in your final submission:
+1. A public link to a **written report** addressing each deliverable and answering each question.
+2. A public link to any relevant **GitHub repo**
+3. A public link to the **final version of your application** on Hugging Face
+4. A public link to your **fine-tuned embedding model** on Hugging Face

Roles.md

@@ -0,0 +1,16 @@
+# Roles
+
+ My Role - AI Solutions Engineer
+
+Here are the condensed roles and responsibilities of an AI Solutions Engineer:
+
+1. **Design and Develop AI Models**: Analyze business needs, develop, and fine-tune machine learning and AI models.
+2. **System Integration**: Architect AI solutions that integrate with existing IT systems and platforms.
+3. **Data Management**: Preprocess, clean, and manage data for training AI models; perform feature engineering.
+4. **Deployment and Automation**: Implement CI/CD pipelines for deploying AI models and maintain automation.
+5. **Monitor and Optimize Models**: Continuously evaluate, monitor, and retrain models to ensure optimal performance.
+6. **Collaborate Cross-Functionally**: Work with data scientists, software engineers, and business teams for seamless integration.
+7. **Ensure Scalability and Performance**: Optimize AI solutions for scalability, efficiency, and low latency.
+8. **Security and Compliance**: Implement data privacy and security measures for AI solutions.
+9. **Stay Updated on AI Trends**: Continuously research and apply the latest AI technologies and methodologies.
+10. **User Training and Support**: Provide training and ongoing support to stakeholders using AI systems.

Tasks/Task 1/Task1.md

@@ -0,0 +1,243 @@
+### Deliverable 1: Describe the Default Chunking Strategy You Will Use
+
+The default chunking strategy that I will use is based on the **`RecursiveCharacterTextSplitter`** method. This splitter divides text into manageable chunks while maintaining semantic coherence, ensuring that the chunks do not break in the middle of thoughts or sentences. It allows for flexible and dynamic chunking based on the nature of the document.
+
+#### Key Details of the Default Strategy:
+- **Adaptive Chunk Sizes**: The splitter first attempts to split the text into large sections (e.g., paragraphs). If a chunk exceeds a certain length, it recursively breaks it down into smaller units (sentences), ensuring each chunk remains within the ideal size for embedding (e.g., 1,000 tokens).
+- **Flexibility**: It works well for both structured and unstructured documents, making it suitable for a variety of AI-related documents like the *AI Bill of Rights* and *NIST RMF*.
+- **Context Preservation**: Since it operates recursively, the splitter minimizes the risk of breaking meaningful content, preserving important relationships between concepts.
+
+### Deliverable 2: Articulate a Chunking Strategy You Would Also Like to Test Out
+
+In addition to the default strategy, I would like to test out a **Section- and Topic-Based Chunking Strategy** combined with **SemanticChunker**. This strategy would involve splitting the documents based on predefined sections or topics, allowing the chunking process to align more closely with the structure and meaning of the document. 
+
+#### Key Details of the Alternative Strategy:
+- **Section-Based Chunking**: This strategy would first divide the document into sections or sub-sections based on headers, topics, or principles (e.g., the five principles in the *AI Bill of Rights* or the different phases in *NIST RMF*). This ensures that each chunk retains a logical structure.
+- **SemanticChunker Integration**: The SemanticChunker further refines chunking by considering the content’s coherence, creating semantically meaningful segments rather than simply splitting based on length. This would work particularly well for documents like the *AI Bill of Rights*, where each principle is discussed with examples and cases.
+- **Adaptability**: The strategy allows adaptation based on the specific document, improving retrieval for highly structured documents while maintaining the flexibility to handle less-structured ones.
+
+### Deliverable 3: Describe How and Why You Made These Decisions
+
+#### 1. **Default Chunking Strategy**: 
+   - **Rationale**: The decision to use `RecursiveCharacterTextSplitter` as the default is driven by its versatility and efficiency. It balances chunk size and coherence without relying on predefined structures, which makes it robust across various document types, including both structured (like the AI Bill of Rights) and unstructured (user-uploaded PDFs). It is particularly useful for retrieval systems where chunk size impacts the performance of embedding models.
+   - **Why It Works**: This strategy allows for better handling of document diversity and ensures that chunks remain contextually rich, which is crucial for accurate retrieval in a conversational AI system.
+
+#### 2. **Alternative Section-Based Chunking**: 
+   - **Rationale**: The section-based chunking strategy is more targeted toward highly structured documents. For documents like the *NIST AI RMF*, which have clear sections and subsections, breaking the text down by these categories ensures that the system can retrieve contextually related chunks for more precise answers.
+   - **Why It’s Worth Testing**: This strategy enhances retrieval relevance by aligning chunks with specific sections and principles, making it easier to answer detailed or multi-part questions. In combination with the SemanticChunker, it provides the benefit of preserving meaning across larger contexts.
+
+#### 3. **Combining Performance and Coherence**: 
+   - **Decisions**: I made these decisions to ensure that both performance and coherence are maximized. The default method is fast, flexible, and works well across a variety of documents, while the section-based strategy is designed to improve the quality of responses in documents with clearly defined structures.
+   - **Efficiency Consideration**: By choosing a performant embedding model and efficient chunking strategies, I aimed to balance speed and relevance in the retrieval process, ensuring that the system remains scalable and responsive.
+
+### Summary:
+- **Default Strategy**: RecursiveCharacterTextSplitter for its adaptability across document types.
+- **Test Strategy**: Section-based chunking with SemanticChunker for enhancing the accuracy of retrieval from structured documents.
+- **Decision Rationale**: Both strategies were chosen to provide a balance between flexibility, coherence, and performance, ensuring that the system can effectively handle diverse document structures and retrieval needs.
+
+
+
+
+
+# Problem Statement
+
+People are concerned about the implications of AI, and no one seems to understand the right way to think about building ethical and useful AI applications for enterprises.
+
+# Understanding the Data
+
+## Blueprint for an AI Bill of rights
+
+The "Blueprint for an AI Bill of Rights," published by the White House Office of Science and Technology Policy in October 2022, outlines a framework to ensure that automated systems, including those powered by AI, respect civil rights, privacy, and democratic values. The document is structured around five core principles:
+
+Safe and Effective Systems: Automated systems should be designed with input from diverse communities and experts, undergo rigorous pre-deployment testing, and be monitored to ensure safety and effectiveness. This includes protecting users from foreseeable harm and ensuring that systems are not based on inappropriate or irrelevant data.
+
+Algorithmic Discrimination Protections: Automated systems must be designed and used in ways that prevent discrimination based on race, gender, religion, and other legally protected categories. This principle includes proactive testing and continuous monitoring to prevent algorithmic bias and discrimination.
+
+Data Privacy: Individuals should have control over how their data is collected and used, with automated systems adhering to privacy safeguards by default. This principle emphasizes informed consent, minimizing unnecessary data collection, and prohibiting the misuse of sensitive data, such as in areas of health or finance.
+
+Notice and Explanation: People should be aware when automated systems are affecting their rights, opportunities, or access to services, and should be provided with understandable explanations of how these systems operate and influence outcomes.
+
+Human Alternatives, Consideration, and Fallback: Users should have the ability to opt out of automated systems in favor of human alternatives where appropriate. There should be mechanisms for people to contest and resolve issues arising from decisions made by automated systems, especially in high-stakes areas like healthcare, education, and criminal justice.
+
+The framework aims to protect the public from harmful outcomes of AI while allowing for innovation, recommending transparency, accountability, and fairness across sectors that deploy automated systems. However, the Blueprint is non-binding, meaning it does not constitute enforceable U.S. government policy but instead serves as a guide for best practices​
+
+## NIST AI Risk Management Framework
+
+The document titled **NIST AI 600-1** outlines the **Artificial Intelligence Risk Management Framework (AI RMF)**, with a specific focus on managing risks related to **Generative Artificial Intelligence (GAI)**. Published by the **National Institute of Standards and Technology (NIST)** in July 2024, this framework provides a profile for organizations to manage the risks associated with GAI, consistent with President Biden's Executive Order (EO) 14110 on "Safe, Secure, and Trustworthy AI."
+
+### Key aspects of the document include:
+
+1. **AI Risk Management Framework (AI RMF)**: This framework offers organizations a voluntary guideline to integrate trustworthiness into AI systems. It addresses the unique risks associated with GAI, such as confabulation (AI hallucinations), bias, privacy, security, and misuse for malicious activities.
+
+2. **Suggested Risk Management Actions**: The document provides detailed actions across various phases of AI development and deployment, such as governance, testing, monitoring, and decommissioning, to mitigate risks from GAI. 
+
+3. **Generative AI-Specific Risks**: The document discusses risks unique to GAI, including:
+   - **Data privacy risks** (e.g., personal data leakage, sensitive information memorization)
+   - **Environmental impacts** (e.g., high energy consumption during model training)
+   - **Harmful content generation** (e.g., violent or misleading content)
+   - **Bias amplification and model homogenization**
+   - **Security risks**, such as prompt injection and data poisoning
+
+4. **Recommendations for Organizations**: It emphasizes proactive governance, transparency, human oversight, and tailored policies to manage AI risks throughout the entire lifecycle of AI systems.
+
+This framework aims to ensure that organizations can deploy GAI systems in a responsible and secure manner while balancing innovation with potential societal impacts.
+
+
+
+## Sample Questions from Internet
+
+Here is the consolidated set of real user questions regarding AI, ethics, privacy, and risk management with source URLs:
+
+1. **How can companies ensure AI does not violate data privacy laws?**  
+   Users are concerned about how AI handles personal data, especially with incidents like data spillovers where information leaks unintentionally across systems.  
+   Source: [Stanford HAI](https://hai.stanford.edu/news/privacy-ai-era-how-do-we-protect-our-personal-information), [Transcend](https://transcend.io/blog/ai-and-your-privacy-understanding-the-concerns).
+
+2. **What steps can organizations take to minimize bias in AI models?**  
+   Concerns about fairness in AI applications, particularly in hiring, lending, and law enforcement.  
+   Source: [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity), [JDSupra](https://www.jdsupra.com/legalnews/five-ethics-questions-to-ask-about-your-5303517/).
+
+3. **How do we balance AI-driven cybersecurity with privacy?**  
+   Striking a balance between enhancing security and avoiding over-collection of personal data.  
+   Source: [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity), [HBS Working Knowledge](https://hbswk.hbs.edu/item/navigating-consumer-data-privacy-in-an-ai-world).
+
+4. **What are the legal consequences if an AI system makes an unethical decision?**  
+   Understanding liability and compliance when AI systems cause ethical or legal violations.  
+   Source: [JDSupra](https://www.jdsupra.com/legalnews/five-ethics-questions-to-ask-about-your-5303517/), [Transcend](https://transcend.io/blog/ai-and-your-privacy-understanding-the-concerns).
+
+5. **How can organizations ensure transparency in AI decision-making?**  
+   Ensuring explainability and transparency, especially in high-stakes applications like healthcare and criminal justice.  
+   Source: [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity), [HBS Working Knowledge](https://hbswk.hbs.edu/item/navigating-consumer-data-privacy-in-an-ai-world).
+
+6. **How can we design AI systems to be ethics and compliance-oriented from the start?**  
+   Building AI systems with ethical oversight and controls from the beginning.  
+   Source: [JDSupra](https://www.jdsupra.com/legalnews/five-ethics-questions-to-ask-about-your-5303517/).
+
+7. **What are the security risks posed by AI systems?**  
+   Addressing the growing risks of security breaches and data leaks with AI technologies.  
+   Source: [Transcend](https://transcend.io/blog/ai-and-your-privacy-understanding-the-concerns), [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity).
+
+8. **How can AI's impact on job displacement be managed ethically?**  
+   Addressing ethical concerns around job displacement due to AI automation.  
+   Source: [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity).
+
+9. **What measures should be in place to ensure AI systems are transparent and explainable?**  
+   Ensuring that AI decisions are explainable, particularly in critical areas like healthcare and finance.  
+   Source: [ISC2](https://www.isc2.org/Articles/AI-Ethics-Dilemmas-in-Cybersecurity), [HBS Working Knowledge](https://hbswk.hbs.edu/item/navigating-consumer-data-privacy-in-an-ai-world).
+
+10. **How do companies comply with different AI regulations across regions like the EU and US?**  
+    Navigating the differences between GDPR in Europe and US privacy laws.  
+    Source: [Transcend](https://transcend.io/blog/ai-and-your-privacy-understanding-the-concerns), [HBS Working Knowledge](https://hbswk.hbs.edu/item/navigating-consumer-data-privacy-in-an-ai-world).
+
+    Do the organization's personnel and partners receive AI risk management training to enable them to perform their duties and responsibilities consistent with related policies, procedures, and agreements?
+
+Will customer data be used to train artificial intelligence, machine learning, automation, or deep learning?
+
+Does the organization have an AI Development and Management Policy?
+
+Does the organization have policies and procedures in place to define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems?
+
+Who is the third-party AI technology behind your product/service?
+
+Has the third-party AI processor been appropriately vetted for risk? If so, what certifications have they obtained?
+
+Does the organization implement post-deployment AI system monitoring, including mechanisms for capturing and evaluating user input and other relevant AI actors, appeal and override, decommissioning, incident response, recovery, and change management?
+
+Does the organization communicate incidents and errors to relevant AI actors and affected communities and follow documented processes for tracking, responding to, and recovering from incidents and errors?
+
+Does your company engage with generative AI/AGI tools internally or throughout your company's product line?
+
+If generative AI/AGI is incorporated into the product, please describe any governance policies or procedures.
+
+Describe the controls in place to ensure our data is transmitted securely and is logically and/or physically segmented from those of other customers.
+
+These links provide direct access to discussions about AI ethics, privacy, and risk management.
+
+## Document Structure
+
+- Both of these documents follows a structure which will make easier to chunk the documents but the implemenation of such section/topic based strategy is complex and timeconsuming as this needs to be dynamic based on the document uploaded. 
+- We could chunk the pdf in sections then sub-sections then pages and sentence/para. This will nicely break the document preserving document structure.
+- There is a chance the user may uploadd a document that is not structure which means going with the assumption that the document will always be in a structured format will not work. 
+
+
+# Dealing with the data
+
+Considering all the above going with a generic approach to cover all uses would be sensible. We will go with usual `PyMuPDFLoader` library to load the pdf and chunk the documents using `RecursiveCharacterTextSplitter` to begin with. 
+
+Would like to use `PyPDFium2Loader` which is very slow compared to the `PyMuPDFLoader`. IF our use case requires the populating the vector store before hand we could go with this loader. The `PyPDFium2Loader` loader took 2mins 30secs to load these two pdf. Comparing the quality of the output there is not much difference between the two. So will use `PyMuPDFLoader`
+
+To improve the quality of the retrival we can group the documents by similar context which provides better context retrival. 
+
+Chunking strategy
+
+`RecursiveCharacterTextSplitter` 
+
+`SemanticChunker`
+
+    Improved Coherence: Chunks are more likely to contain complete thoughts or ideas.
+    Better Retrieval Relevance: By preserving context, retrieval accuracy may be enhanced.
+    Adaptability: The chunking method can be adjusted based on the nature of the documents and retrieval needs.
+    Potential for Better Understanding: LLMs or downstream tasks may perform better with more coherent text segments.
+
+Advanced retrieval techniques tried
+
+1. Context enrichment - Creates some duplicates need to investigate why for later
+2. Contextual Compression - Creates better response but takes time. Will need to check if streaming helps.
+
+More for later.
+
+Experimented with above chunking strategy and found that `RecursiveCharacterTextSplitter` with contextual compression provides better results. 
+
+
+# Choise of embedding model
+
+The quality of generation is directly proportional to the quality of the retrieval and at the same time we wanted to choose smaller model that is performant. I choose to use the `snowflake-arctic-embed-l` embedding model as it is small with 334 Million parameter with 1024 dimension support. Currently it is at 27 rank the MTEB leader board which suggest to me that it is efficient competing with other large models. 
+
+====
+
+Your response effectively breaks down the documents and uses a clear, methodical approach to answering the task. However, to enhance the response further, consider the following improvements:
+
+### 1. **Aligning Chunking Strategy with Context**
+   - **Current Strategy**: You mention using `RecursiveCharacterTextSplitter` and `SemanticChunker`, which is a good start.
+   - **Improvement**: Since both documents have well-defined sections and are structured (NIST RMF includes clear sections, and AI Bill of Rights is principles-based), it would be beneficial to first chunk based on sections and subsections, while combining it with context-based chunking. Instead of focusing on one chunking method, you can adapt based on the structure of each document.
+   - **Dynamic Chunking**: Also, mention how the method would dynamically adapt to less-structured documents if uploaded in the future, ensuring scalability.
+
+### 2. **Specific Chunking Examples**
+   - **Blueprint for AI Bill of Rights**:
+     - Principles can form separate chunks (e.g., *Safe and Effective Systems*, *Algorithmic Discrimination Protections*).
+     - Subsections can also include further breaking down examples or cases cited under each principle.
+   - **NIST AI RMF**:
+     - As each section (such as "Suggested Actions to Manage Risks" or "GAI Risk Overview") has detailed subcategories, chunk them accordingly.
+     - Include how you will preserve context when chunking specific actions.
+
+### 3. **Incorporating Expected Questions**
+   - You have already listed good examples of user questions. However, for the purpose of improving retrieval:
+     - **Enhance Contextual Retrieval**: Suggest tailoring your vector store to group similar questions by topic, such as data privacy, bias prevention, and AI safety. This allows better retrieval of relevant chunks across both documents when users ask questions.
+     - **Example**: A question about "data privacy" should retrieve answers both from the *Data Privacy* section of AI Bill of Rights and the *Data Privacy Risks* section of the NIST RMF, creating a more comprehensive answer.
+
+### 4. **Document Summarization in the Vector Store**
+   - If possible, create summarizations for sections and topics within both documents and store them in your vector database. Summaries improve quick lookup without requiring a deep scan through every chunk.
+
+### 5. **Advanced Techniques**
+   - **Context Enrichment**: Mention that it needs further investigation but is a promising avenue. Focus on eliminating duplication by refining preprocessing or filtering steps when enriching.
+   - **Contextual Compression**: Explain how you might use this to generate concise answers that retain meaning, which could be useful for long or dense document sections.
+
+### 6. **Handling Duplicate Content**
+   - Add a comment about how duplicate information across different sections can be handled by maintaining a cache or reference of repeated content in different chunks to avoid redundancies.
+
+### 7. **Performance and Efficiency**
+   - Since `PyPDFium2Loader` is slower, clarify that you will use it only if high-quality, OCR-accurate extraction is critical, but `PyMuPDFLoader` is your preferred option for efficiency and initial loading. This could be useful for streaming applications.
+
+### Enhanced Structure for Response:
+1. **Problem Statement**
+   - Continue with the problem definition, but expand on real-world implications of ethical and risk management in AI.
+   
+2. **Understanding the Data**
+   - Break the two documents down clearly into sections and discuss specific strategies for how chunking can preserve meaning within these sections.
+
+3. **Advanced Retrieval & Chunking**
+   - Expand this section to include the chunking methods you've outlined, and specify the improvements you will explore (e.g., dynamic chunking, context-based grouping).
+
+4. **Performance Considerations**
+   - Detail how you will balance quality and performance based on user needs and document types.
+
+This would strengthen your approach, improving both the technical accuracy and user experience.

Tasks/Task 1/pre-processing.ipynb

@@ -0,0 +1,1299 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "% pip install numpy==1.26.4 \\\n",
+    "openai==1.44.1 \\\n",
+    "qdrant-client==1.11.2 \\\n",
+    "langchain==0.3.0 \\\n",
+    "langchain-text-splitters==0.3.0 \\\n",
+    "langchain-community==0.3.0 \\\n",
+    "langchain_experimental \\\n",
+    "langchain_qdrant \\\n",
+    "langchain_openai \\\n",
+    "pypdf==4.3.1 \\\n",
+    "PyMuPDF==1.24.10 \\\n",
+    "pymupdf4llm \\\n",
+    "sentence_transformers \\\n",
+    "langchain_huggingface "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "BOR_FILE_PATH = \"https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf\"\n",
+    "NIST_FILE_PATH = \"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf\"\n",
+    "SMALL_DOC = \"https://arxiv.org/pdf/1908.10084\"  \n",
+    "documents_to_preload = [\n",
+    "    BOR_FILE_PATH,\n",
+    "    NIST_FILE_PATH\n",
+    "    # SMALL_DOC\n",
+    "]\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stderr",
+     "output_type": "stream",
+     "text": [
+      "/Users/jeevan/Documents/Learnings/ai-engineering-bootcamp/AIE4/mid-term/ai-safety-chatty/.venv/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:13: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
+      "  from tqdm.autonotebook import tqdm, trange\n"
+     ]
+    }
+   ],
+   "source": [
+    "# Embedding model - snowflake-arctic-embed-l\n",
+    "from langchain_huggingface import HuggingFaceEmbeddings\n",
+    "\n",
+    "model_name = \"Snowflake/snowflake-arctic-embed-l\"\n",
+    "embedding_model = HuggingFaceEmbeddings(model_name=model_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pdfloader import PDFLoaderWrapper\n",
+    "from langchain_experimental.text_splitter import SemanticChunker\n",
+    "\n",
+    "\n",
+    "pdf_loader = PDFLoaderWrapper(\n",
+    "    documents_to_preload, PDFLoaderWrapper.LoaderType.PYMUPDF\n",
+    ")\n",
+    "documents = await pdf_loader.aload()\n",
+    "\n",
+    "text_splitter = SemanticChunker(embedding_model, buffer_size=5, breakpoint_threshold_type=\"percentile\",breakpoint_threshold_amount=90)\n",
+    "\n",
+    "chunked_docs = text_splitter.split_documents(documents)\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Recursive splitter\n",
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "recursive_text_splitter = RecursiveCharacterTextSplitter(\n",
+    "    chunk_size = 1024,\n",
+    "    chunk_overlap = 100,\n",
+    "    length_function = len,\n",
+    ")\n",
+    "recursive_chunked_docs = recursive_text_splitter.split_documents(documents)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "\n",
+    "os.environ[\"QDRANT_API_URL\"] = getpass.getpass(\"Enter Your Qdrant API URL: \")\n",
+    "os.environ[\"QDRANT_API_KEY\"] = getpass.getpass(\"Enter Your Qdrant API Key: \")\n",
+    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter Your OpenAI API Key: \")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['8dd5b1e7fd464e2a90c28a8eea8b0cb9',\n",
+       " '906e0c268d564dbc89c0b8398e235407',\n",
+       " '4b81191a4cc94fbd835dc9c942e9543a',\n",
+       " '25c3b7fffa8d4bc29790057fe2f4d025',\n",
+       " '3ad5906a8a274b56bd05e4ac39ffe459',\n",
+       " 'e3fa01bef57c489ca014be2e589b7ef1',\n",
+       " 'af5fc5121c6a438a8fc5dea454b7e92f',\n",
+       " '80500cf02d5748c39b1c62288459c306',\n",
+       " '5db6eebee14b4aafa948e4f9aa4f7aa2',\n",
+       " '99385298e8744643822e01525bdff89e',\n",
+       " 'eddc9704820d4005b7c62a5085f69454',\n",
+       " '4324a624f4054ae5baa7270d9f6aaa56',\n",
+       " '9eb24bea31a749f1b7a86ac2b186ec14',\n",
+       " '7e9c9763bebf40cea1833ea6ad376eeb',\n",
+       " 'cc8846008cac472e88eb16497c560a15',\n",
+       " '5af0886e387449fc89f1d0e82c32c590',\n",
+       " '824ae7c1c15a43c8b62713f02d91e0b5',\n",
+       " 'f0ef1b30251b4429ad7d902b85fafcf8',\n",
+       " '314a75e55d1b4c1fa46f49610d745f95',\n",
+       " '66828a5f9536480bbd08d94f087bc44b',\n",
+       " '8230b8add982486f9ac8e120a27d3aec',\n",
+       " 'dd1c75bb5c1441468ac8e7d4595bf0b9',\n",
+       " 'a9b1b1b87eeb48b78ed4cf6adddee9d2',\n",
+       " 'eeacab16c9d94d08a791c516e0a65f6b',\n",
+       " '187badb4dc064743898f5e5218114250',\n",
+       " '0ecc4e873fe047ce8afc33e19fe40c3f',\n",
+       " 'be7b81185ce140229bee6d1306120528',\n",
+       " 'd8a9a361dc8a4917aefc2e0a17efafad',\n",
+       " '7ca3b3bf947e449e8f58cd4fca12d884',\n",
+       " '07a8b46151e74641b97ad823ef91082f',\n",
+       " '421004f00dbb4a47a81c424ad5f64e39',\n",
+       " '53750f6e3cfa481d971d7e5a6b9f55c5',\n",
+       " 'e9f9524be6884599893590ad5acdb12b',\n",
+       " '446225894d9747e0a888b596875ac83b',\n",
+       " '7677215528c44061bfb018e42a13e528',\n",
+       " '183002bfeeaa4b75968dda61451c2f37',\n",
+       " '818cd888dd6b4735a602949aea2ea900',\n",
+       " 'c803c1690b5549a5a13578b2cd757b17',\n",
+       " '864bc201486e42bdb4caef6a266fa1ee',\n",
+       " '02cae44fcb9e4d2eb60a6da08b01a4e6',\n",
+       " '384e2a4f36d14359b22b0c0cdf463cd9',\n",
+       " '6de4fdb915164aa68e076a2d6e1913ae',\n",
+       " '8da0e4ecc526416ca3be4043879ea17b',\n",
+       " 'e63359693473494d922f996b57c65d3b',\n",
+       " '76313322b20b4e368f2aaefbf911df6b',\n",
+       " '455bced7b2544b69a5c547d998548748',\n",
+       " 'd1abce93130d4eb49cdc1aca8b7c9c60',\n",
+       " '966dde23cfa144b899b60a3659f32eb4',\n",
+       " '4fe01c77f1ae4b70a18c56fc6cb9679d',\n",
+       " '7aa14bc014fa445cbc061c47a3fe1c31',\n",
+       " '0aa076f900614fcfb37883057c67e6c1',\n",
+       " '2bae6b24f8234792b914a75712fb89fc',\n",
+       " '74fb5f5dfdb8413a9afaaf472a009ac2',\n",
+       " 'b1c95e4126e842d598eae6fbc455b82c',\n",
+       " '2e326893157844fe88f762b96aa46b99',\n",
+       " 'd27b5c85573d44658a4d338c39890629',\n",
+       " '1f967a5cae05418d94f9f3d07dcce74e',\n",
+       " '7667f8b8e5914417a68e7d41256cfc98',\n",
+       " '7fcd90b390494d3686e532a6528bd021',\n",
+       " '8c64aecd850b48b3bac216e73ebad1e0',\n",
+       " 'ca845f98d6b44c02b9f9edafcd75aa2e',\n",
+       " '571b12484be44ea6b406e6cdcd0662a5',\n",
+       " 'fa099bf314614075bf8bfc58110f52f1',\n",
+       " 'b98bfe7e8c234ed59b5d893929bac64e',\n",
+       " '3bc58c54665d4a4ca394ebb13debfabc',\n",
+       " 'a276d45eccd54056a8855ccdb5907df3',\n",
+       " '0f9ac79d967942d0b56b9fe200dc7846',\n",
+       " '07c5e3794b244e28b384dad31a2c63a2',\n",
+       " 'deb29326ff1e4fec934c35710d4e0dcf',\n",
+       " '5278dac4122044879b9dd1a7c557b7a8',\n",
+       " '041338e04daa4482a7d65c311ab0f3f5',\n",
+       " '1ed9bbc381b1423a95ef935cc16e277c',\n",
+       " '20bb221a9b654bbc99edc812475adc12',\n",
+       " '188fba73978143e8a22370774e1d31fd',\n",
+       " '3a65a41271a947848aba4939473d0f85',\n",
+       " 'c5cee035027048338a81b9bf0830cf57',\n",
+       " '03325dbfeb164512a565172407ae0ec3',\n",
+       " '4494b4f19cff440281c034ede5e675e1',\n",
+       " '2b55dd78b0f5461d917eeaec2a75dcb7',\n",
+       " 'a2aef7ef0741438fb643a4fb225f1ffd',\n",
+       " '3886d787efdd4d2bb3fc702ffd911db8',\n",
+       " '71455bc57478429e8a1269ce7332302a',\n",
+       " '0c2896f473c749f9888b5723ee834a5d',\n",
+       " '1a8507e2597049dc8287d21172dfe518',\n",
+       " '943e65c033774aed99116750cfbb5f5a',\n",
+       " '78bfa949218f48c7b365569f2c3396ff',\n",
+       " '6812e197478a40bf86cc363d11fc0856',\n",
+       " 'e9cd2bf8ad454aa7af9446e62d9d845d',\n",
+       " 'ca970fdebb1e4d79853090a7c73722f5',\n",
+       " '84400697300c4d468f5c58d09fd63d56',\n",
+       " 'ad1bf5c566a147e6b66f9f3502a227fd',\n",
+       " '0c620c86587a4ae8975a0d066eb80e97',\n",
+       " '1aa1ebd384774410ac011ca3e535808f',\n",
+       " 'f30f7dba0182402c8abce8d9b07df99b',\n",
+       " '73dd6c906bd14d3494dc8def54680e0f',\n",
+       " '632c7eafa1c048979b65c5e8ecacb98e',\n",
+       " 'b3edcc98879747dbbed2e8b6e19e1baf',\n",
+       " '4f4bef9639bb454e87cb61051e2d4c82',\n",
+       " 'd0c059bc4e04474f9355c87964b3b470',\n",
+       " '412fa6deb62546c4a092988f96ddb425',\n",
+       " '2283f3ef3e7141738c8966fa4333ba05',\n",
+       " '7ed4340667d643b193b45f7f21a238ed',\n",
+       " '9209830e86c54dbd9974cc737bbdbe91',\n",
+       " '89e28024e5f14377a6b2efa1997f370a',\n",
+       " '2f4b769a22b24bb49bed8917adae1f9e',\n",
+       " 'fcfc743d434d43b886afb80c5377e1fd',\n",
+       " '300f3cfca0874ee2818241856f2175df',\n",
+       " '55ea80ccf78f4c9cb622c1451951e723',\n",
+       " '4b755a511dbd46f79eb4b3bda119e79c',\n",
+       " '956a7f2f70854e1a82dfc542fd761492',\n",
+       " '190c9cd3f01a4ea894877f4ab35000bf',\n",
+       " '381a0acf443244c78f303b9f6b72535f',\n",
+       " 'd7afcec2075343b19d1320605cc41b46',\n",
+       " '7898f81259be4c42b44cdfd3b41aa25b',\n",
+       " 'd86e65ae1f8140e299ccf27583735b7a',\n",
+       " 'ee553712557545a0b0320adc4d563bca',\n",
+       " '6fb6a0b739a64b909cee096d722a4f6e',\n",
+       " 'be1514e798f04581af33447b86f002c8',\n",
+       " 'c92113c2d4344ce8a10e7a6c1d089f4e',\n",
+       " 'dc47dd1e29bb49768c2b88054f91c69f',\n",
+       " 'd72643a36bf6415795a3694f93e5c376',\n",
+       " '0709e83c2d974cb7aa30b17f8a5e5050',\n",
+       " 'f4bc420a5b4c4cdd91c4441837fbcfae',\n",
+       " '712e9898ff5f44eda54aebcf54931760',\n",
+       " '1916383ddb404a32ab833c2add8e2511',\n",
+       " '4a01c037505943a3be4fa183de7d5c73',\n",
+       " 'cb29c514ccf8416491654441749f9889',\n",
+       " '57cc735133754979862b8dd27ccf45d8',\n",
+       " '6e28a749e0ee49d8b6ba562ab268e474',\n",
+       " 'd915b593c4194759a4ca48304ce54b56',\n",
+       " '83bf4923ef4847b1b3faa0a85ce85d9c',\n",
+       " 'c7165907a5774d7a9cfe034328875f16',\n",
+       " 'fb83c16fe2ac4a0b8a2cdd3a099b0751',\n",
+       " 'd8d56a63f4104e1d9b06c5c8d6246d4d',\n",
+       " 'b92d479d4fea486980ce1133ef0d9049',\n",
+       " 'b264b04721c14738a6e018c3d089e3e2',\n",
+       " 'bf50880e770d4e2a80415e87b8f95788',\n",
+       " '5ca16b29007f4b919ac1f3fdf261aa10',\n",
+       " '30f3d89d1ae042afb3b745451e0a5fdc',\n",
+       " '41dd324a662a4e79935980dc8e53ab8f',\n",
+       " '7590bcc7d6d540b1bd92a7ce69c0e9b2',\n",
+       " '72973a23774d4bebb9c42dcf885ae06a',\n",
+       " 'ce3e692e73084116ae834e72349032a7',\n",
+       " '044469614deb404f8d3b1860907e0f75',\n",
+       " '3805bb3205c5411daf2a64a7742e59fe',\n",
+       " '03aa772e62b44423b75ec05c90e8687e',\n",
+       " '740e824d876d44c7b30599b4dcb8eb44',\n",
+       " 'ba27340cf2144d15bee2a5f5b7e00622',\n",
+       " '4c6a7847bc554fddadd0a884c26612bc',\n",
+       " '8a13a8d664c4453b8f71c01b28ec8dff',\n",
+       " '113a3db0cd0d4e0f8067c5fa074967df',\n",
+       " '28af19cb148f49049336aa1b52c14a98',\n",
+       " '795bc0bc5a9c4ff8b472e2a9c9c59dbd',\n",
+       " 'fb36fe1dad1c4280a7186ea5c20e64dc',\n",
+       " 'a50e1d30a67b4144bd8ce5ab32f1cd3a',\n",
+       " 'f1ccd9cc27b0414f96243f1c63a07fd5',\n",
+       " '3d2b64f6ceb74744b6b8374728142334',\n",
+       " '15f314cbb8a14f9286a814cafef76192',\n",
+       " 'ac540651b7a34d50b70e4c44cf25b3ca',\n",
+       " '6b87356c50d1404abe0a676b7f322a72',\n",
+       " '67902b525d014249900e54257590f7ea',\n",
+       " 'fc1189d79c824a74bd60dd5dc341aa2b',\n",
+       " 'b26845eaa60246399cef48b0a13d11b5',\n",
+       " '7b592acd329743a8a7a3b2569a048416',\n",
+       " '189e134f601441cc9f1514a778e3c820',\n",
+       " 'f7f1425e4c2d4e1fa9040ee85d368bf1',\n",
+       " '0297a5233d6f4275baf0a9957b0dc586',\n",
+       " '5afeb076d11841c890517fc92d0aa6f9',\n",
+       " '79fa6ed91f7a47b6bd764e1c8b412fa5',\n",
+       " 'ff4b75e4daaf4588ae69ba2f83816c15',\n",
+       " '84b7e45334a3477f8d8a64e3504fb620',\n",
+       " 'dbf780a26828491da830425df5a7a03c',\n",
+       " '25a6912b64f442f99f5787bef114ebff',\n",
+       " '00c07386007a4dc18072a431f7cf83b1',\n",
+       " 'f84828d74c0c446389732b8eb4d6570f',\n",
+       " 'a12ba2aaf84640a8816d9ce8e8a417e9',\n",
+       " '02223a887b2c4ec0891d45e75836b00e',\n",
+       " 'a62632e5379a4af5be885b4750d18650',\n",
+       " '5ef2d149eb314d879897648027e7e8aa',\n",
+       " '1a2c86d6906141b18700239300599566',\n",
+       " '584530895cc74af58cdb016c0ed63bc5',\n",
+       " 'e19a1e82a1ec4884a7c72f2996ac927f',\n",
+       " 'dbcd348813cc4365bf65c549333e669d',\n",
+       " '8bac146f886b4272bd40f51adb35c32b',\n",
+       " 'cae2438601594d6fb39d99d617fe6c0e',\n",
+       " 'e8306f6a959b4a219d096b968784c44e',\n",
+       " 'ebbdefa7da15403294655048c6fe3624',\n",
+       " '60579bd40852405b8345114456963981',\n",
+       " '211f66d20b5c451d91f310594b854ea0',\n",
+       " '6d726194ac8641a6a5f6d8ce3f192a7c',\n",
+       " 'c0154e0f56b049048f9bcb7f718173f2',\n",
+       " '081a6179661e41f69ab10b92027d161e',\n",
+       " '2c149c226d504053bfa94532a850efd1',\n",
+       " 'f833250f67bc4329922a5a7f7b7d07f7',\n",
+       " 'a0c1e7c49351406ca3567622b6b1e38e',\n",
+       " '72abd1e5dc824d6c8852f7331990b6f0',\n",
+       " '35c02c07ae484045a325940fcbe098fb',\n",
+       " '63b83f297d1d486e84ddb42c2af32900',\n",
+       " '17fb247ed7bc4599a8de06966e744b2a',\n",
+       " '371ea02f2a7e4e11b82ed0593a26a806',\n",
+       " '6847a89d7a944bf2bd95430c4d63def7',\n",
+       " 'a186a88983e64831bf42523b6522d706',\n",
+       " 'e3538719cffb4cb59efa815b27b4bf81',\n",
+       " '8ae7bdfed4e249dd98727cbb4f34259d',\n",
+       " 'd4f18e23e8444ffba7b13661b22ba1a9',\n",
+       " 'ab0aace578b2457cb10966f9a57dbfce',\n",
+       " '641437d62c1940d7a7d0711391802aa6',\n",
+       " '3f88fe5f258244d581a053a53b844bdc',\n",
+       " '5022a64b46dc4989b2a919193cdec7da',\n",
+       " 'e620e702fb604457b4724ce0f753138d',\n",
+       " 'ff9557d7d3c446a39829de03605a5254',\n",
+       " '5314c1de556d4b96af06fd3ac37cf1f4',\n",
+       " '56d98670e8f74b1881bac44ccfb9267d',\n",
+       " '2d06bf2cf37a48f98708e345e86a6114',\n",
+       " '1d6be38e70b74ce69cf68c32fdef0b9f',\n",
+       " '936184dab9ab4ff9905cbaccc0844e61',\n",
+       " 'ec080c630727417fa858340935e0557d',\n",
+       " 'f6792c2c58774d4e9ba97575ae5a9ddb',\n",
+       " 'c1df1df5060e41899363e2a0649100e4',\n",
+       " 'b2eb147898c64b359c951297318e6831',\n",
+       " '216d56dfdaae4c098b826c2c6dbe8132',\n",
+       " '7921cab4d11646168b5d186794f5db24',\n",
+       " 'b89e6d297f064e708c4fb903c6ebf15a',\n",
+       " '4ee471c58dbe4185b6968113228bb20e',\n",
+       " 'd64a74ed5b57427c9c6ce98a9f945b70',\n",
+       " '58cd9cbd849c456e85fb72a4abc5c69c',\n",
+       " '2e1835610aa749c896c8c165e3d84470',\n",
+       " '8008134cc1c44751bb95a3270cb89a44',\n",
+       " 'e4f098d6b9024392adc396aad0efb94f',\n",
+       " 'e8fcb91812d048efa5ba38a46cf40531',\n",
+       " '987e19cc2d674e2aa0d555af45ef874f',\n",
+       " '2816810a2eea4f0081baae4b28614796',\n",
+       " 'ffd0647c27664a779dabf843fcf83981',\n",
+       " '7a5e718e26b14f4daf674c901b3dde93',\n",
+       " 'a7248347d0b743d7b5db65f3b1b87cac',\n",
+       " '640f3399c6c340f19a11919a6402ba85',\n",
+       " 'fe73657d7e884bd48d93afdc193b73bb',\n",
+       " 'a9c649a44de94b5f82af06c804e3bb08',\n",
+       " '8265df5e7847443a8c91478ae1cabf72',\n",
+       " 'a3bb9ab12f814c4faa382b586fe13680',\n",
+       " '775f072a48674d6b9fdd0671c4064891',\n",
+       " '21048806b4c74f5db5b7f873c45adedb',\n",
+       " '1525157eac174ab684089f50e6c29969',\n",
+       " '136132cedfec4e3bacf2a8adc1fbd50f',\n",
+       " '08099904461749ddba96b17b61226622',\n",
+       " '1da17dd87dd448c4902f8e3a1ad1c51e',\n",
+       " 'fa9c4880b4a34b9aaca3f5363ff1b7f0',\n",
+       " '30eeafb17ae74a42b370e173e22abfea',\n",
+       " '0d3ad50f8c524e90b6c440865aaf63d8',\n",
+       " '51d52addb9df4c04afaf8f008fe89259',\n",
+       " '2aa50c713fe241f1b9b44107c0d47945',\n",
+       " '8615b8c9442c4031aee25316cdfc7cb5',\n",
+       " '8f8fd8a1edc043ee82c77381bf39a83c',\n",
+       " 'f15223b9a98445f2b7613e518e7bca83',\n",
+       " '6a437e8d6655430aaef679f6c6a84831',\n",
+       " 'c0a3331686754cd9929c4abd5d81dc7f',\n",
+       " 'ff36401f33d9424cb7112033ead5f58c',\n",
+       " '841220093a1242f0b04a4ac8d852e280',\n",
+       " '655ec122f3d24c069eecacc8e8bc8f82',\n",
+       " '6da4281d97ad46ed9ef6dd169c640afe',\n",
+       " 'eeef919024d54063ae3cd6c6f8f7a73a',\n",
+       " '9f55e4818e0c4bef82dec178dc64eaa4',\n",
+       " '8a652387c6eb424288a0022be058d00a',\n",
+       " '9a19d1114e674c618d23a1299f14f1ba',\n",
+       " '9e4b5fdbbda24ed5a2fbbc3923847a44',\n",
+       " '8b442e6de7bf401b8300c567a642a759',\n",
+       " '20c8152b25514d018eeb8542b4450ad2',\n",
+       " '5ae8d5d8230f46cab713cecbd97c847e',\n",
+       " 'e159ce134b7b46308fc919b22a9e808f',\n",
+       " '5a0d312175db4d15b85c0255b68bc027',\n",
+       " '6f82139d091145cba88a7f0fc367063d',\n",
+       " 'ee60e7e10d924f01b530b0291d939aae',\n",
+       " '3b73fe9ad953458bbd3d11f44b85fce5',\n",
+       " 'd15e56f06ba24e3cb6a1c4dd0568201d',\n",
+       " '16f3d27489ff423dbf7d027844d957db',\n",
+       " '6091bda6320149a1af5ccfb541e75148',\n",
+       " 'ec527d7203164f07ae7349cfa33829b3',\n",
+       " '26b4090286e346b4b686b13360cceea3',\n",
+       " '856dd023a8cd41108eea38b403eadd09',\n",
+       " '5434f50e81db44e5b80d3bc8816eb5bf',\n",
+       " 'e9022b1d39f24ec09981e8c66478705d',\n",
+       " 'd3f895478ed74239b4bc88e04e215f1a',\n",
+       " 'c1fb1bae1b5e42f8a65f8260c259e133',\n",
+       " '2a86b477a07e48afb2658742c30494e8',\n",
+       " 'da64e968fc3047089de2ffa4b62a8c0a',\n",
+       " '0a42ebaaedae4f73914398ad1486afb4',\n",
+       " 'dc66a40fba5f4e348216910fce0d2428',\n",
+       " 'e125ed2241b24a31ba40ed768a21d4f6',\n",
+       " '850eda324b734ffdad0fe63c92c91038',\n",
+       " 'd9e9ca7b0a634afdac1f4da62f2dcddf',\n",
+       " '7cf441e12164420fa8b58e0aa6d244de',\n",
+       " '0a6c2f48e5094e3399f7e1d0f38d873f',\n",
+       " 'c8c29dbfc7f840d7a7195aa74388b30c',\n",
+       " '07a90e50dcbb4352baa6636e9b687aab',\n",
+       " '85d60bfe6d684c1a8578c1d6710c867b',\n",
+       " 'ceb46b27e8994626a6d6d1c1acabff5d',\n",
+       " '506dd325656145ebb7d976de3b4953c0',\n",
+       " 'a83e954196874363b13c7cb3d7d8e025',\n",
+       " '16fc61ea959d4427b3fa723d7e58f2bf',\n",
+       " '3d4599e2ad2f47deae8c1c25d30dec68',\n",
+       " '260482de224a4ec998459a5d2f9384bd',\n",
+       " 'd480305e9fc34a55b8f146343fe1dd8e',\n",
+       " '9851b805fcf54766bd482d5a0d4a8d0f',\n",
+       " '5147fb1a9a904ff09b7c6885567fa94e',\n",
+       " '509f1c4ef2b348af821461d751850e93',\n",
+       " 'a045c48dd9444211a2f0087229df189d',\n",
+       " '3b90d03add21451aa40990b1f2dad9e1',\n",
+       " 'fa68102a0555422db1cc0f3822496a48',\n",
+       " 'ade010ff9a2644a38c7c3de875a3ac78',\n",
+       " '390f54300e1f41ac9224da683f00d31a',\n",
+       " 'aacaee53fce14e9395259a0609cc1646',\n",
+       " '00ca0c3998b64339874ad036983a0922',\n",
+       " '2b3b851a8bb6422abab843dc2148255f',\n",
+       " '363ed4276aa144b2baeec9dbc1fba38c',\n",
+       " 'e9bf60fe2f184793b37f268ffa486abc',\n",
+       " '98b185639a5d441ca60d7a5fe7620f8c',\n",
+       " '9ae478f07ca3465a9a447b3c7eab4b26',\n",
+       " '01fc2f0676754dc7baae898343e2bebf',\n",
+       " 'cf3dd2f39b1a48919b90571555e4befb',\n",
+       " 'b0ef26aca0404662b5706ccfa737a52b',\n",
+       " '9afcf7171eb74e628f99fa44a753c131',\n",
+       " '26f3917cc2274e998e115212273fe2ba',\n",
+       " 'ec369428b5fe43138d049f293dcd21a4',\n",
+       " '947d581cb3cd4555933504b8c64c54d3',\n",
+       " '16b495459a4e4bfd96a12655defa9551',\n",
+       " 'af15b6c943834e96a24363fbdca209f3',\n",
+       " 'a01a9d5cb1c041889bfd1cf29cd4c08a',\n",
+       " 'd3d0ee8f3c394ed680324fdcdb442241',\n",
+       " 'd56148720f974dffab53a4e8917c3833',\n",
+       " '32593dde550f45af9a36349bcd63192c',\n",
+       " '1d43c4caf83b470897e96410f4dae5ee',\n",
+       " 'ee1cd8b353cf45e3a88ce76faeebc9a1',\n",
+       " '8e7135cd24764e94b8d04e15ec86b9c0',\n",
+       " 'a7a918cf594e4a2992398acc924e6015',\n",
+       " '13837b8571154abe83bb0b8d8e08d406',\n",
+       " '63a4ce49d82d4da6bcb4da66db26bf35',\n",
+       " '2214b1db8432499286a9ad49d8a2391c',\n",
+       " 'af23e4413b7c42cc982b011d6432ec5a',\n",
+       " 'e4b5e669227c4112aec7a7c53f568b75',\n",
+       " '2d5f634bb7414afcac7b78ce7c0a864b',\n",
+       " 'beda5449b3124e379f35601a33ab4651',\n",
+       " '271af180d99846e4a0d8c57f444df81d',\n",
+       " '4af16168d5a1432e8ca9719c9000f58c',\n",
+       " '405113fc9e334cada56589b758cd9fd7',\n",
+       " 'c95f295e46ba4b82b9f92fc0dcc8c1df',\n",
+       " '475897fbe33347cf907f3cc381f40c0e',\n",
+       " 'b6779e2220c444d38741c06cc2bb380c',\n",
+       " '7d56c936c7d84514a67cd75e369449f7',\n",
+       " '272a9892cdb742dcbe5f90e29eefae72',\n",
+       " 'c28ace207c9d437da68cf599ac028bbb',\n",
+       " '6d3c684dd6894bd9bf24486175ed834d',\n",
+       " 'd8766ca5bb7d468399e6b864756a04e6',\n",
+       " 'f5501aae471447fba9a4ac7ccf88c1fe',\n",
+       " 'bb4d8f7876a141e0ba82eaebe7899c5a',\n",
+       " '1c87be78b3fd48a093c23a54904bf8bc',\n",
+       " '7cfe5d24a86645e1928a4700e2175e82',\n",
+       " '79a9e904f5bb48a3830647c6afccbb85',\n",
+       " 'deda349c16f54a9f85cd302269c22456',\n",
+       " '5339992d8dcd464294260f5c0c857fff',\n",
+       " 'd86ca1d2c07f4784956acb34d4d8c48c',\n",
+       " 'dadf0561bb1c4ad9a87cc33a21424d32',\n",
+       " '63c86b1adaed4514a75e0409a66b15c3',\n",
+       " '61425b5443b840f2b7d28347d4002192',\n",
+       " 'e7166ad200694bb7ae645e63495dbfa5',\n",
+       " '9a4e61507dbd4fcd96b9c4b8eb24e74d',\n",
+       " '5cb628ca8b8245e0ae326ccb8ae5635d',\n",
+       " 'cbf3322896f8445ead83a6907a9aae08',\n",
+       " '9156196800e64996891c0703499ffbb5',\n",
+       " '47c88e62be7e4cb88b7d4935ba38cff0',\n",
+       " '0fee655d64c34f84a07e6b889866a486',\n",
+       " 'e4472727736f4fa59d49536d8e331f95',\n",
+       " 'f933c36480f64d8b9600c5075a085e61',\n",
+       " '808c6b3ce87345b391843aaa6b253bfb',\n",
+       " 'f5d6ee781bd742fc88d5ddc2e5f0a7f4',\n",
+       " '281ac17550864cf5827193ddd577aad1',\n",
+       " '3d3aa0ebb1574fa7b498a13abb1b7c40',\n",
+       " 'f43bf31cfe994208b24e363f9459a7a0',\n",
+       " '9931894cfd004a20991a7fef40c23c86',\n",
+       " '1b34b9f61f164993b7387a73e961bf2b',\n",
+       " 'e2ca8df6b02d4240b7f1e4474b4765c7',\n",
+       " '42741aaeaec7422f8ab9c59d18430455',\n",
+       " '6a0a67f326704e11baae384eb567fa09',\n",
+       " 'f7894024f0764978a9eca821c29d3449',\n",
+       " '7f88bf5493764642a14a5bd8bbf04a71',\n",
+       " '93a5412c61204d53b94fda693fb561b8',\n",
+       " '3d265760e45a45d990240628c46fde6f',\n",
+       " '62112a36dcab48379590ca210ca09918',\n",
+       " '1abadc8684e64e6c8cbe1f7427d39678',\n",
+       " 'f1f8c1a6c7534c5bb386425728cfa2b3',\n",
+       " '81b640e8ae0747daadeda29da9f677f6',\n",
+       " '6428b2a89f384a6985d69b0183fc71c3',\n",
+       " '107fb02d9c7e4bffa9669509015e8af1',\n",
+       " '73b8fd9b8aea45a6adfc02d5795bec62',\n",
+       " '931cdf852b634abfb01b656221a8d0ae',\n",
+       " '81e9f8177fe9430a9fb17fd20522c955',\n",
+       " '7dbc18b381454afeb2a6041f60c2b23b',\n",
+       " 'f621fcf8f34f4629909ca455ecaa4f55',\n",
+       " 'c358be2860cf436d8fabb3200888c307',\n",
+       " '81516460f65740e9aef0f4babc29b2f1',\n",
+       " 'ecd2409c27cf47aeadeff569bd25ea85',\n",
+       " 'a4c997a1ad7f4990b2d71cb028463610',\n",
+       " '1ed8f8421791456db0543cd3e1ede40f',\n",
+       " '56f5077192e74e09a58017d0c3368bc4',\n",
+       " 'a639e85d3bee4530a53d132bfa7c58de',\n",
+       " '4c75146c59dd4541a8500f89dd060a2c',\n",
+       " '95c74438067f4bada1fee37942e06ed7',\n",
+       " '0b3f010515574c48b02bebf7a451052e',\n",
+       " '4db576ae022d42beadd921a81e977096',\n",
+       " '3998f2de8bf44929afa7ad0e2e86eccd',\n",
+       " '73cf1599b76d4061874e660228ca5f06',\n",
+       " '962fb1291b984d60adb133201b7eae48',\n",
+       " '365a19df65514c698d826e86fcdc6091',\n",
+       " '5e86e99df25a4cc287d8ea0605f8cb08',\n",
+       " 'f2cccab55efc43d5b098c38c31f687fb',\n",
+       " '3388b8f7db314bf5a60cd10dbbc45f9c',\n",
+       " '2a83e0ed7b4e4d2f906cfbc8dca7c512',\n",
+       " '5b4f99c2acab40248de70a0e92506bc0',\n",
+       " '278560e5a9e244e1a0a2ffa0ef7c261a',\n",
+       " '864b65e24dea4473ad0e4a5bc32f4c69',\n",
+       " '6b089bf4dd004ed78f1b92c50d414e47',\n",
+       " '5bebbff6685649b99fa304d40b9b6362',\n",
+       " 'aecb11bcf1444ad589508ea8bec77bdb']"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from langchain_qdrant import QdrantVectorStore\n",
+    "from langchain_core.documents import Document\n",
+    "from qdrant_client import QdrantClient\n",
+    "from qdrant_client.http.models import Distance, VectorParams\n",
+    "\n",
+    "dimension = 1024\n",
+    "collection_name = \"ai-safety-sr-arctic-embed-l-semantic\"\n",
+    "qdrant_server = os.environ[\"QDRANT_API_URL\"]\n",
+    "qdrant_client = QdrantClient(url=qdrant_server,api_key=os.environ[\"QDRANT_API_KEY\"])\n",
+    "qdrant_client.create_collection(\n",
+    "    collection_name=collection_name,\n",
+    "    vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),\n",
+    ")\n",
+    "\n",
+    "vector_store = QdrantVectorStore(\n",
+    "    client=qdrant_client,\n",
+    "    collection_name=collection_name,\n",
+    "    embedding=embedding_model,\n",
+    ")\n",
+    "\n",
+    "vector_store.add_documents(chunked_docs)\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "retriever = vector_store.as_retriever(search_type=\"similarity_score_threshold\",\n",
+    "                search_kwargs={'k':10,'score_threshold': 0.8})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 44, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': 'b6779e22-20c4-44d3-8741-c06cc2bb380c', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='Human-AI Configuration \\n'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 33, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '26f3917c-c227-4e99-8e11-5212273fe2ba', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='Human-AI Configuration \\n'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 11, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': '07a8b461-51e7-4641-b97a-d823ef91082f', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content=' \\n \\n \\nFROM \\nPRINCIPLES \\nTO PRACTICE \\nA TECHINCAL COMPANION TO\\nTHE Blueprint for an \\nAI BILL OF RIGHTS\\n12\\n'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 37, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '1d43c4ca-f83b-4708-97e9-6410f4dae5ee', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='Human-AI Configuration \\nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV \\n \\n'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 61, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '73cf1599-b76d-4061-874e-660228ca5f06', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='et al. (2023) Whose Opinions Do Language Models Reflect? arXiv.'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 28, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '00ca0c39-98b6-4339-874a-d036983a0922', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='Make sure these tests cover various scenarios, such as crisis \\nsituations or ethically sensitive contexts. Human-AI Configuration; \\nInformation Integrity; Harmful Bias \\nand Homogenization; Dangerous, \\nViolent, or Hateful Content \\nAI Actor Tasks: AI Design, AI Development, Domain Experts, End-Users, Human Factors, Operation and Monitoring \\n \\n'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 59, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '81516460-f657-40e9-aef0-f4babc29b2f1', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='https://www.rand.org/pubs/research_reports/RRA2977-2.html. Nicoletti, L. et al. (2023) Humans Are Biased. Generative Ai Is Even Worse. Bloomberg. https://www.bloomberg.com/graphics/2023-generative-ai-bias/. National Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \\nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework. https://www.nist.gov/itl/ai-risk-management-framework \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \\nRisks and Trustworthiness. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \\nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \\nDescriptions of AI Actor Tasks. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Appendices/Appendix_A#:~:text=AI%20actors%\\n20in%20this%20category,data%20providers%2C%20system%20funders%2C%20product \\n'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 57, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '1b34b9f6-1f16-4993-b738-7a73e961bf2b', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='(2020) Overcoming Failures of Imagination in AI Infused System Development and \\nDeployment. arXiv.'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 0, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': '8dd5b1e7-fd46-4e2a-90c2-8a8eea8b0cb9', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content=' \\n \\n \\n \\n \\n \\n \\n \\n \\n \\nBLUEPRINT FOR AN \\nAI BILL OF \\nRIGHTS \\nMAKING AUTOMATED \\nSYSTEMS WORK FOR \\nTHE AMERICAN PEOPLE \\nOCTOBER 2022 \\n'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 23, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': '8c64aecd-850b-48b3-bac2-16e73ebad1e0', '_collection_name': 'ai-safety-sr-arctic-embed-l-semantic'}, page_content='Some companies have instituted bias testing as part of their product \\nquality assessment and launch procedures, and in some cases this testing has led products to be changed or not \\nlaunched, preventing harm to the public. Federal government agencies have been developing standards and guidance \\nfor the use of automated systems in order to help prevent bias.')]"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "retriever.invoke(\"What steps can organizations take to minimize bias in AI models?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['dd370438231c41dbb7b1b4f1e7673cf7',\n",
+       " '02ebba25e01941849b9e2c9d5097b55d',\n",
+       " '099f0083356a4914b53fcb30df633b50',\n",
+       " 'f8aefa25a4544c869ca4caaf686b3d47',\n",
+       " '9ec0798fb4554f95ab65bd05315af118',\n",
+       " '33bdad4db0ab4145b85726f77f1789ad',\n",
+       " '98a75a601b114b07953b5aef4e032b4a',\n",
+       " '1e49952c0d6743ba8ad52a049c18daa3',\n",
+       " 'c3babb9205e54ca99ba6e5a03679bdba',\n",
+       " '74cecdae132c4a5e953bd7e72ac6850e',\n",
+       " '29529ea9530541a0bb446a8e82fab913',\n",
+       " '4193dcf34f6249b1a29c49a52239deef',\n",
+       " '84cb5d0f2cee47beabd72baa54161155',\n",
+       " '622f279ac5bd40b082725d90972e9ae3',\n",
+       " '48e366f92aa449e89cf7158584d2cf6a',\n",
+       " 'e2ffb7cb2ac3482fb9290940fabe9582',\n",
+       " 'f52a4c3353544fff93f241cba063028a',\n",
+       " '0c81aa08ddd4496a9aaea4b001f3596c',\n",
+       " '3e9d8d7785b04d5fad063219c94ef0dd',\n",
+       " '76796785c7b64d428e48b7cf699e155a',\n",
+       " '593ab20fc2494634959b0bfd8821ea91',\n",
+       " '654421ae91df4739bfb1ebdfb7c9dda2',\n",
+       " '27ffe059aafd4d5fa795b2f893b1d57e',\n",
+       " 'f1468d8276444858acb33bd6e2d36e73',\n",
+       " '5a6a15255cdd438abd9b2c3358dca939',\n",
+       " 'fbb13ef430ca47d28013dda9feaf4625',\n",
+       " 'fc16826ddd504038bb5f32fd97cdd98e',\n",
+       " '72c878c56d8746dea51fdcf506e48894',\n",
+       " '257ac1e04a4b478ab3b84c81e5dfc3f4',\n",
+       " '68b157c05ced46828ac39894e69b8d08',\n",
+       " '535e59df03184e86b30a09cd2d169dcd',\n",
+       " '1a6d76252d364a758564a41b922d44a4',\n",
+       " '61e497e66868447988198ba831096707',\n",
+       " 'dd9f18bedfed443c8bed0fc4c34c5e23',\n",
+       " '0ca0575097c24b50a613d5a19de61cfc',\n",
+       " '7dfba6cbbfe34756ba3f40b1be282324',\n",
+       " 'e9b68e9579194b04ad65bbf85332d351',\n",
+       " '7545cee6d2e345ba90e95082a15271b8',\n",
+       " 'df1e9db6843a4ddbb788b1e9117db9a1',\n",
+       " 'bb0687d2f3d047138d0414d0b2a22917',\n",
+       " 'b79ed7024a064c1f9360692c93615657',\n",
+       " '70fb8aa096a74d0a975705ac44f08577',\n",
+       " '41bf93d83ebe414e91253e7a96f50ec7',\n",
+       " 'bdab13de5b514bf68921751b3051ce60',\n",
+       " 'ccd47e89a09c4519981dc5d9be7b1ad9',\n",
+       " '334ac2db387848f1829e174c6584288b',\n",
+       " '5484df8c41cb41babd01c3f8d62121a2',\n",
+       " 'dd1f97aef70e439ea02c8f0d0ea397e0',\n",
+       " '99ccff600f8b4470af445f1f060e5518',\n",
+       " 'd4de01e6623741d3b06c8ee973ad6670',\n",
+       " '6217b664cbba4a64bf6e4f2ffde27831',\n",
+       " '3974c50b7e3a4503925f1c397254d259',\n",
+       " '4959c05e7d8049a4b75cff3bdc6fc30d',\n",
+       " '9d3aeacd6513463fbe9d13c1fb2441fc',\n",
+       " '8777904b546e4ef5b2759f0a60fd1fca',\n",
+       " '7a73c81712804111b6145b57888455ae',\n",
+       " '87036e89882546b69613378b17610332',\n",
+       " '5c508cd4449449c486b811d65b9b6db1',\n",
+       " 'fd5a25bb9038481aabaed2b34a7f2cc9',\n",
+       " '80b47526b0224fc0ba54cf4a61da11cc',\n",
+       " 'a2c5d0697278407fb0d89c9c138bfba0',\n",
+       " '49cb7eb52ea043f2baf21b611709d83a',\n",
+       " '96d002f0aa0a4cd7a86b188ef7811e9e',\n",
+       " 'd5a4fd354f904b99a8700363f7bcec7d',\n",
+       " 'ec384dfa0a5d4caeab593a4d013e40de',\n",
+       " 'b613bc8f681141249d11f2eea7691f32',\n",
+       " '7b9e491ad88b48f19ce2698c4d8ef5ec',\n",
+       " '011946a9fed74b14b7d4be2ff4eaadf5',\n",
+       " '2c07a4769e85425a9a32a053f1293ff2',\n",
+       " 'c859a58fd5a54447a22217564e610e77',\n",
+       " '2d6fcf19e009459e82b344a699c6556f',\n",
+       " '22ec0544ffc44be1bd557cd91e96caf4',\n",
+       " '1a9fed777bc8454faed8c60a12dce190',\n",
+       " '898d974afb3f472a96c9cca3c698fee1',\n",
+       " '21cb161a28f34fc89b90b369d1895fd6',\n",
+       " '9409c811bf6542feae01351580bcb32b',\n",
+       " '1219e011429b4e29a84268bdbb66d7a3',\n",
+       " '3293d7aeecf54778a4b1e63f09f3f362',\n",
+       " '543f56abfcda469a826e797aa2a4ae36',\n",
+       " 'b714fd2157d145658860f0db0bd95163',\n",
+       " '8be2abe5fee54507a03f0a5d0ba2f0c7',\n",
+       " '76bf633b2d6d49b98845023de4024f09',\n",
+       " 'eb00c39f80904f79847f0156e1e88ff0',\n",
+       " 'c9695522dbb243cbaaf48b9a5b9f4105',\n",
+       " 'aa3a41bf64fb4136b1fc097ad40378eb',\n",
+       " '4ba6b441be194008aac5fec9aa0eac53',\n",
+       " '7b5e6c78bcc64d4c879cc0817436ab35',\n",
+       " '90229474433449648237b410db3cdfb7',\n",
+       " '1ace5952f4c043e0a0864b9926475add',\n",
+       " '2208ad4c34bc4fdb940e1cec9df0f6bc',\n",
+       " '15e5cc7345b64cc08ad9825085af5486',\n",
+       " 'ccac823288be4ed2a3d5617dda575120',\n",
+       " 'e01e349952e54ef4849519acdaa6725b',\n",
+       " '794cb0ec54724738a48d16e18f6cb3b9',\n",
+       " 'd9ebc60044124f0a890f7836cb58f4a2',\n",
+       " '9fda4771fa994fcaa609088cfc961dd4',\n",
+       " '67fd7a926392436a9b344903bbbc08f7',\n",
+       " '7892533c0b57466e8249f11d5cba07d3',\n",
+       " 'ba1b7ad3addd4557aa6983cc309fdd49',\n",
+       " '3663e44b29e14192abe4f49c98e3db45',\n",
+       " '476728c883404adea36c903128c98139',\n",
+       " 'dbec4491b8b94aa0b9327ead46b4251f',\n",
+       " 'b635ebf0a3644e71ab64bfd43faec517',\n",
+       " '149ee2b23af9448c896c5bf87f1c9257',\n",
+       " 'f6082f115738427db840c3ff58a7c48d',\n",
+       " 'd56f7d290c184a5aabdd80251ee807b5',\n",
+       " '37738bf3df9048c381a26116632aff03',\n",
+       " 'a139a0ccd8834a3293c65b8a9fb0a2ad',\n",
+       " '6cce500b28a944fbb6ad637ae5d3c227',\n",
+       " 'e16444a94afb409b827c3ee3f57237b9',\n",
+       " 'd2d0cd1eac154f17b31a6a97d87bca88',\n",
+       " '75bc3a67451e44a487939e6ca74e39ad',\n",
+       " '9074393c760844908a39e806f7d4714e',\n",
+       " 'fe16063331f848d897ac4466e9237fd7',\n",
+       " 'a9504de4c7cd427cae397ddd551e3bba',\n",
+       " '791979c2b3d747bf857370c4ab7e7757',\n",
+       " 'cc31a5edfbfd486482b202b8b87c8e9f',\n",
+       " '6cf68d7de2334461809f1867dfef1280',\n",
+       " 'e8acd67d2b614ebd8e0786dcd961c05c',\n",
+       " 'e30854b096ab48b3b7f3729bd07914e1',\n",
+       " 'ae46462add43450d899655e2e0819e59',\n",
+       " '1718ec5c859c4b37a8c8ac0bbcffa616',\n",
+       " 'ae4a9b112f364a34a47f0cf255b882a4',\n",
+       " '5ab58a4ae877437898e06968149195b2',\n",
+       " '76401834e26049b096cdbb054cb37c7a',\n",
+       " '52b9f9683ba14fb2a6f84f5b6d619b40',\n",
+       " '96d56773bf2e4d858f62654f20ccd53d',\n",
+       " 'd8a258c7815040808dbab44252e15e77',\n",
+       " '9f4ca5df27674f4cba5628dd98f1ef2b',\n",
+       " '8e0679ac4cbc44839a7f77a12ce52220',\n",
+       " '964aa8b0653c4bdaa236d5bb6f1eff1d',\n",
+       " '219697ddd2d545c484443ca116943b63',\n",
+       " '607bf4aa92c64da9a4873477e2e9b363',\n",
+       " 'efd6114026d2454d983e7b4063656266',\n",
+       " '81fdf32f6fb949508196043e4142261b',\n",
+       " '14f82f6a28a9488a93ce3724bdc2a476',\n",
+       " '3d44caaa301046af9acdd238d1e3cbda',\n",
+       " 'd8d11ff667ed4c0f94958e794e9b2c60',\n",
+       " '90ba7f9ad22d46d99316b36cf213ec01',\n",
+       " '76e251b11fb04240ae381af227c136bd',\n",
+       " '3dec921799b24369a5b9189dd28c0f55',\n",
+       " 'e7d706e31326405aa1a8acaa627ee2f6',\n",
+       " '9b8c00a9f69649a1a66decee2aa77c9c',\n",
+       " 'e2902a1405cf41db9a0428259089cbef',\n",
+       " '5d43e9e802fb49098531b257c5633723',\n",
+       " '413aeaa9a2bc4970b21371940026dba8',\n",
+       " 'bfb4471ff2af4bada2ac8bcd14429d24',\n",
+       " '533ff7a97fca4578999e6e0434df17d5',\n",
+       " '04ba353da230424bbf25ab29a18e20f4',\n",
+       " '1ed6dc9440f14ac9b1deadc10fbb660b',\n",
+       " '734aa1cc14a34825be7dbc947ecdb525',\n",
+       " '482153d0c05f4453a3bc9f57a1804406',\n",
+       " '76d988efad614d3dafc0ecb8fbdb2189',\n",
+       " '6d9ada3651704a0ca2b40664a59c8579',\n",
+       " 'ac9272192bfc416d8386487c9b381ccf',\n",
+       " '0d8c4c52de304f7f9778455f8ad178ec',\n",
+       " '9daf6aed58624240b7fb1fcf79d9dda2',\n",
+       " '7ee8e6bfacca440f9ef345e59eba7401',\n",
+       " 'e80b50ed3a3a42d2a24bfe85cd5d45db',\n",
+       " 'c0a0384aad9345f3b6162bf63c5bc0d3',\n",
+       " 'df7c71db74af415b96cb2cea32d0ba30',\n",
+       " '329b6e0ed9cb43cca48970fcc286e299',\n",
+       " 'fd4c170eb78d401fa3eec3253be26b98',\n",
+       " 'a98ea19a24cb4b04bd943d226dc41bba',\n",
+       " '8e917ca54af2400cbd83e08e28ef0bb0',\n",
+       " '78a3cf3647cc44f9abb083eaa8b79947',\n",
+       " '140a48e4acbd44da9fede75e12fa80f3',\n",
+       " '92d6e0856f5848869f7feb8cd17d7088',\n",
+       " 'bf482ac0daea44a789182b52fbd2f413',\n",
+       " 'f562e2d3c5b9422b845b6f87806e4d6c',\n",
+       " 'e8b0bdec066a4bb0b7a47ec7e10c10e1',\n",
+       " '45a054f0a9824867a52db472b2b65ad4',\n",
+       " '40ff751a666347ed9c2326341587ea51',\n",
+       " '1e35fbf001de48599063d4fe6dae165b',\n",
+       " 'fa3b6f2ebe274d54851e9a31975470a6',\n",
+       " '42242351bcb844c589f44a80cc139fdf',\n",
+       " 'e5eb97e74797481d998677225cbaf365',\n",
+       " '85f4bc0ed08c49feb6fd69f5659eaa36',\n",
+       " '87df9b4b3b8a44eaa8756bbf8c967d8e',\n",
+       " '7c04f74911aa4d629f0a545155e60b8d',\n",
+       " '133597b368564215a8b71b2535a07032',\n",
+       " '97473d57f4bd40af96f63bc06a1c6117',\n",
+       " '3b3f2e2c08774f42bf0c904230b06c4c',\n",
+       " '9661e9d3901e429d9030b04d28d98a19',\n",
+       " '488e7a22bcdd400fb1ea9e52a102cd8f',\n",
+       " '95086a933034484b9eeca08343c0dc21',\n",
+       " 'e5aa0c58bc1448169bdebc7e99fcbd42',\n",
+       " 'e6ba2f1ca8284d11a309757bc42ead7f',\n",
+       " '6510e8c73d10408fa038b242f95cae2d',\n",
+       " '36c2ba7197ce4c238472599a256f60db',\n",
+       " '12ce3404c5024cc5a3ca4d1c0773d759',\n",
+       " 'd0b909588d804b1aa89b496efcb6d16f',\n",
+       " 'e830a0b4f0fa483b9bb9816162bde54f',\n",
+       " 'eea7403faa024fc8a477b4b2e12bfc99',\n",
+       " '1fbbcd091c3948229841d1a1e53cedef',\n",
+       " '0bc14c99eb8141fc9f5ad1a13e8c5f90',\n",
+       " '544d8570f3e847aea814771f3af2397e',\n",
+       " 'd4e66a38f60c47deb7307e0f65829409',\n",
+       " 'b520c61605aa473d89c88e3d277f40c5',\n",
+       " '3f01a965995e495a8b3067fd0fdcc978',\n",
+       " '9a5da84235c14817a3f0bda30a2bbcbb',\n",
+       " '94792a1b19654f45a2fd8cc362dacddf',\n",
+       " 'b46df9b21e764b39a0180bf42f9a835b',\n",
+       " '47a4942336cd45cea7afcc68d99f1cd5',\n",
+       " '2081541601de49199f881ffcb1625d4d',\n",
+       " '972dcffe6c2e4cf2815cd571e9f4021d',\n",
+       " 'e4613bb436fe46aaa3c236a209038124',\n",
+       " '355cc289660d494db52e039127ecde34',\n",
+       " 'b2b84fffde454e2d967ec6330e637b37',\n",
+       " 'ab32b49526ee485d998dca366dace258',\n",
+       " '48e0545b903f49bca771a22861166708',\n",
+       " '47f62df12d494ba48b1bfee4bc1820f0',\n",
+       " 'c131bfc970324e068ff4e04df6191c8d',\n",
+       " '02a961ad52a34d1ead8d5c1a9ee12031',\n",
+       " 'dcc8094b71444056a9c85f1b69b7e6df',\n",
+       " 'd95234dd8571413ab9dc2cf2bd4031ef',\n",
+       " '2b80007625d04c3a8b3c10fd35181861',\n",
+       " 'b2b150d718c64b38925de1ee0abf14ed',\n",
+       " '49f5645c803b4da78f09c7f0d337867a',\n",
+       " 'af890841867746499efe8600704630b4',\n",
+       " '7218a0d2f3e34a729da8a10e41a591fa',\n",
+       " 'bafcc1d1244f40da99adcfb72f87b170',\n",
+       " '14f2b87359524bfbac74a4948fdd135d',\n",
+       " 'b209da5ac4ad4110834f018a3301f5cb',\n",
+       " 'bf06ddade01d466592ea9cadbada320d',\n",
+       " '865d5986afde44f4ab593708125e90ad',\n",
+       " '3be18dd8e0bb4940bc58b257bad9c5b9',\n",
+       " 'c02da0af95774b39af650dc268c8eed0',\n",
+       " '7fae32ee9e934e2b8f164212ad9190b0',\n",
+       " '8d0c1b678ea742cca445577d36a58e26',\n",
+       " '4c1667fd01804d08bca4485a427b7cf3',\n",
+       " '5d3bf9345565447095a9ffd9319997d9',\n",
+       " 'ca5889dd43c5498ca449a733c36631d9',\n",
+       " '5ba45f56b2f0412c835d4328b88037d3',\n",
+       " 'e1d3f4649f234a8395a63ae1de670449',\n",
+       " '17e91fcb2ae14f56a0f60c6acaf4258d',\n",
+       " 'ed1d8496e014462db3aae0a046d4aeed',\n",
+       " '8dd4483ba29448b7a285cefcaeb135fa',\n",
+       " '77db975e07284a7a814ef386664c97aa',\n",
+       " '93c28d0a7eb646969d0511b786fa7a71',\n",
+       " 'c596f3b0927c49c1a4193eb5f0479395',\n",
+       " 'e1648b2975284446bbbaefba431cdd78',\n",
+       " '76bcc756b84a4b169a128973ef7228fe',\n",
+       " 'ba57f715ffaf477fa15a733ddf5339aa',\n",
+       " '644902133d4645bfbd02d9629fb737da',\n",
+       " '124a4a8ec036421486e8501be3af4692',\n",
+       " '0b95233f27f54043aee48dd77096c62c',\n",
+       " '1ef09ede43d546b4a6a73b48c4cb48f1',\n",
+       " '9dbee7e4bb32427f8fd0b0229ca0d2a6',\n",
+       " 'f915f126e0f24f2299e6bfd16a5d3c1d',\n",
+       " '3686b153c85248f6a2fc1fff12eaafe3',\n",
+       " '850e99ca1e58439c8ccf36e2b6a7ecde',\n",
+       " '25c84a37812b47c8adfd41b30af8c0bb',\n",
+       " 'f96eb4a5818e4ebf8ae654d35cdc08a4',\n",
+       " '31aaf38fa0bf49f4964d317f000840fd',\n",
+       " '8c914f8f496741dab0d661f8bf84e061',\n",
+       " '215a8d37eb5249ae97b9471c8ec0f888',\n",
+       " '9f5e41e99d314cba824998d58ca1a611',\n",
+       " '6b01b294c1774e34919059a7388aabd9',\n",
+       " 'c1c6025360a9458085d5342cf8e703e0',\n",
+       " 'b3fdbd3082794bd4b0ab4d4f2c8149f8',\n",
+       " '700188e4d52b44fa9fde7512f54d7b1b',\n",
+       " '00dad32bd08f4b39b153cc96b8497f4b',\n",
+       " '9bb6f94103404153b68855d9993e9493',\n",
+       " '56e07e044da34280830555b42799444d',\n",
+       " '1d57812e47de41bf99efbbfa34865acd',\n",
+       " '39c2def37c7d4b15a4e766632ea9eb98',\n",
+       " '93e308150c2b44688cd13847402815b0',\n",
+       " '1f595875c8ee4ac9a261bfb0a429067c',\n",
+       " 'e7043593429e48a6bdcd8095f3ee2993',\n",
+       " '8827c0bc83eb4dbeb48befa28e6ded29',\n",
+       " 'acb099517d0449239adf6c9dde626772',\n",
+       " 'f94e571e12af4903bfe866f6e028124e',\n",
+       " 'd027a4d37b3640ad894587dab59a7494',\n",
+       " 'f35c824977cc4d46962b01ad10f5ceb4',\n",
+       " '836279f1552c4417a642da79743aeb33',\n",
+       " '002978f8f7ab4cada169ea0d054499a5',\n",
+       " 'e7a1c0978a2a4cd0b6b317ceda9874fc',\n",
+       " '0f977a8cf0514392882756b1f7c6fa26',\n",
+       " '7682cce87cfd470d95274b61e4eef8b8',\n",
+       " '3947f0e87d00475387486b47326ed258',\n",
+       " '6ddc01005056438cb611cd958b7a2d1d',\n",
+       " 'bc81516df0f440d8a1faa7363f011b75',\n",
+       " 'd676819f49004a56b7dc89cc5d5343ec',\n",
+       " 'b8fd60c1b629499dac6ea2cdeb837502',\n",
+       " '4e33d391a5634f0d82dd84dda6957811',\n",
+       " 'b007e62d2fe749a6ab713c005211a73f',\n",
+       " '18de6dda198e4f36befe7c81f88a7f42',\n",
+       " '029857b9fae04c498b62b46c40267afa',\n",
+       " '3be29a9773f24c079f24d9db9c662801',\n",
+       " 'a095542002ce46ca95e59094332f0228',\n",
+       " '40d61bdffba64369af605108a12a2999',\n",
+       " '7edeaefb56544debb539a1bafb766796',\n",
+       " '2f5335dc55594a04b67b07d86d937139',\n",
+       " 'f89a5a9d2d5047d7a74dd991ae0e8102',\n",
+       " 'fd4b6403ba4249b19c692a9dbdcdba01',\n",
+       " '3e02d74c82764b6997c4f965cfd6c233',\n",
+       " 'dfbe54ff42b4457db6cf921dc4ca0753',\n",
+       " '7abb841a671b40649ad478bc45c75b47',\n",
+       " 'a687e1b6e9dc40099a7d7d4ecd021a46',\n",
+       " 'ffdea99f5cec44e4abc9f4b8c6949fc1',\n",
+       " 'cb103ab7aa1a46f09d858dcf6880c862',\n",
+       " '4709ecfdce6c4392a245fad38093d1c4',\n",
+       " 'dbceabef7601444ba3a76c6bed960802',\n",
+       " 'a7f1acab40b145cbb2d8d84fc72733af',\n",
+       " '4df54db71ffa4dc4b1f10b022e3e6ef8',\n",
+       " 'a7fc14d6dbb14af6ae3ef0a3f68f1d07',\n",
+       " 'd5048b397eb04c6fb97a083b66aa6ac2',\n",
+       " 'b6420e8387b94f85aa36b5bfe589463f',\n",
+       " '767857ebb1fe41269ed4d82d967956d9',\n",
+       " '400246f61fa94320a267b7ab3f2e8cc7',\n",
+       " '8b763f3947974bd192d2d884c05c6428',\n",
+       " 'ea8f1f4852d64e09be5ac90b04404dad',\n",
+       " 'f6866c404c2c40b3a4314563846a911f',\n",
+       " '837631a58af84142b5042772b24da3bf',\n",
+       " '9d1586da31e44e6f9fb64ce9ff157673',\n",
+       " '098b85fd43c24f1aa3fd7c93a48fb98b',\n",
+       " 'f324b2ea7aef49979b23bb34f78846e8',\n",
+       " '6c35050aaecf43e7b7fcf40e9edfaa2f',\n",
+       " '910792f650bb463493a1b85488133ab5',\n",
+       " '2d428a2e50194db792e5a02146be9364',\n",
+       " '90ba4b5f05124c26aaec626e14ff2138',\n",
+       " '5c4b11be82e54ebab89237c6a4928284',\n",
+       " '11c9ce7a16094f788485a13218479435',\n",
+       " '27ef6c0ee62d4f4d9767d255f5b0bda6',\n",
+       " '603495b3760e453bbc7264b287754bd6',\n",
+       " '2afe6de14bc34f6381f30124a069d391',\n",
+       " '8666745692804c2bbd16b997d75f9426',\n",
+       " 'f9789e9010d44d2fa2eeadd121f9186b',\n",
+       " '3fdd94668daa4bd6a96e125c2725d9b4',\n",
+       " '02cf5c44de76414bb431468c721ea6ad',\n",
+       " 'ea3dc743311b46c6b7c6117c57de0333',\n",
+       " 'cac324fb19374fc892f4db768850823d',\n",
+       " 'ab4126594df44beb906274bbb1c0f40b',\n",
+       " '4cb686862a9a46ecaaa414860edba1cd',\n",
+       " '0e6481c6b52e43e5856a779b814b509c',\n",
+       " '73c0feaefcd44cffaa712880076005c7',\n",
+       " '147e1975a5b545c39eedde3c9e112d3f',\n",
+       " 'b52a309d67e44730a3e13f395aec79d4',\n",
+       " '6068165a21b64de191a203024d30275b',\n",
+       " '939af554456e4f9cb33268bd36d792c5',\n",
+       " '3dffe1eb87754b6ab3c932d8d77cfa00',\n",
+       " 'bf7419dc8ad84fc9b15e09b9125fe6b8',\n",
+       " 'f76718fe634243029f02130498d5afcd',\n",
+       " 'ef23c33828ff408abab8607b82eeb016',\n",
+       " 'da5f19bce12048a2aa11b06b85072f9c',\n",
+       " '32f57a9a758a4e41849fd85cedef76f0',\n",
+       " 'fba4c4a802904152bbeb6edb051e2607',\n",
+       " '728acf0b65ad4108a1e7a72b146e338e',\n",
+       " '0ff6d22870074917a6c014c33f4b7cf9',\n",
+       " '29d7bb91ce6b4a74893c614f725c5178',\n",
+       " 'b690e48b7dff464cb73dbdf1e6149309',\n",
+       " 'd92a821e7fdc4276b4fc8201dadfef62',\n",
+       " '7dcaeef50440471f83e8febf47b1049c',\n",
+       " 'd06797d35176423f97a118fb2921bf35',\n",
+       " '8601101c7fa9428c9624bce8ea4cee15',\n",
+       " 'f45441e8e3264add88478323c93e2d38',\n",
+       " '985d73938f0744fd9ca3e05d8ed4d99e',\n",
+       " 'aec96ecb915d47eea4ae0dae9dc95446',\n",
+       " '8120edfc5cf247babcaaf6e7bf59ebe1',\n",
+       " '821cb61115ef45f3b24ecf7e7ffe5b27',\n",
+       " 'e4dc1ed27a224c9294f648336d261c53',\n",
+       " 'b90241dc87844d628f45401349ab9887',\n",
+       " 'a255e1c4b31241058595c134f5de807a',\n",
+       " '648c5ad4530442e5b50f28f40c386bb6',\n",
+       " 'ffa4b3e9e4ae4289b0c404da4abadcbc',\n",
+       " 'fa5af409bfe441d1bcee2d0b5e377678',\n",
+       " '033a5c2bd8374ca6b8cb9b4d965954ab',\n",
+       " '357babd6dd4947749142359d0fc0cdd3',\n",
+       " 'e4853735544842de9790530cb56a3eba',\n",
+       " '3698a84c5a8644d99072de0b3a6aa9f5',\n",
+       " 'd9c6bac4f1dd41dcb3318843c7f79489',\n",
+       " '3dd799500efb450f8004d4240e037b20',\n",
+       " '686426621d894b2781e6fb48d4b16c8e',\n",
+       " '85b807e001ff46fe985770fa6af9a534',\n",
+       " '097971e5d47043a2b4d569d56634bb2e',\n",
+       " '46bb945a79c94a7da13c0c1506e1c457',\n",
+       " '5ef5cbe182d94b5d899d2dbc9595b3a8',\n",
+       " '466985e3a3ce424ab38284db938f8d40',\n",
+       " '3a87f5d24ee448649a1fb37e1572f0e4',\n",
+       " 'e17801c99d6944b899da6568724826a9',\n",
+       " '24d0a98cbe0d4450a1e3994c6dab3a15',\n",
+       " '3fa0add1d0b642f296690f408b0372c8',\n",
+       " '1d657774ca004dc79b7fcb36ac85e26e',\n",
+       " '7e69a8dc03104c72bdfc8cb6c2fcf9ee',\n",
+       " '40c85cbead07447a8ba67f4c279ffd8b',\n",
+       " '7b595edf61784a549e64edfc1e18a497',\n",
+       " 'a4d50cd2c1534c02b0a34458d25920d0',\n",
+       " '318e51c677b949d09d9a61fb7a069082',\n",
+       " 'a45a8c5d6a6a47ccbafb57a1bb45c4b0',\n",
+       " 'a0c089e55b15476e8a89292b31b310fc',\n",
+       " '41fcfb134a8a44c9931edccd36627ca2',\n",
+       " '7e7687c8087a4174850cd19935c845bd',\n",
+       " 'a812d2adb03546538480ad44b33fd2bb',\n",
+       " '5c5779f29b93468ca603bf37687d068d',\n",
+       " '50783f0c8c944ff19aa86f2e5ac781ac',\n",
+       " '4a21f30d19f24fd5a331537371b46dce',\n",
+       " 'adda51e0076048ca98142173498f3af7',\n",
+       " '07fa53932ec041d5bcd71d77b273d8d2',\n",
+       " '18768f62094043548f4d280627a9d3a9',\n",
+       " 'd5fcc7eeeb154b179028b03beaf8f3f8',\n",
+       " 'a6859c0e9eb74acfbfde7fefbc76d9b6',\n",
+       " 'c2e90d49e7f14233acebd2ac10622efc',\n",
+       " 'e0997f7ee41742ed8a0179f9805bf12e',\n",
+       " '9b8dc4963513406d90a71935b05a7601',\n",
+       " '1cfc3fe1c73f43f69d776c880641baa9',\n",
+       " '34fc9ec282fe475d8314ea0a3a44b881',\n",
+       " '0943841a14ef474b8844520d91bfee9c',\n",
+       " 'ef27e818cad1462aa3bf7d9aaa19700f',\n",
+       " '6e2fdf8601904078986ebb1c71bc8168',\n",
+       " 'c18bf0d5ee5e45c8806cc1ea7d486bba',\n",
+       " 'a87b5e1ac2dd42d488929580625996de',\n",
+       " '8d384236c7e448439e9230a000d6aaf7',\n",
+       " '6dabbe7ff4814ab4bdcb84715ab20af4',\n",
+       " '943a13d4ef0d4bc2a81b16078b580e78',\n",
+       " 'baa6fd13ff6f445a8adc6482e59eb411',\n",
+       " '57b39b4eee4d4a0aac587d53ef68ff8d',\n",
+       " 'fd07301c3d554f02a0793d5f7bb63f35',\n",
+       " 'e99969118642471f887206e1c6a507e7',\n",
+       " '584dc05d60844536bdaac7bb3c1b7cd4',\n",
+       " 'f1ba0bf7f2b54612a3ffa4a66dd0989c',\n",
+       " 'e254c679f27c4143bcdda32f15e846e1',\n",
+       " '2d51c45ed9ca4a3ab95590cff7047d37',\n",
+       " '3bc9228d85364bd1b1f5bb7d13af5a8c',\n",
+       " 'b2de27d51d964d8c9e190f42d7ad9768',\n",
+       " 'e08419ed7e154a7da62745b3bd5ebd78',\n",
+       " '70a099122b2d43dbbe23d375432beaa1',\n",
+       " '01c551bc80134225ad0391bb295b365c',\n",
+       " 'c2388131208c4d8c868af7d7e6405cca',\n",
+       " 'bb0e3a6a10cc4ac29a168fcee5042c17',\n",
+       " '4b20b4b550da419d88ee62758c495138',\n",
+       " '5ea09fb9fb074218b814dde11c1aed3f',\n",
+       " 'ee62a59cd0784f4cbabb349725d7fe78',\n",
+       " 'a58457312c084595ab30bd5c59d0b3cb',\n",
+       " 'af5f60ae6a3a48129b15c89bebe493cf',\n",
+       " 'e5fddd76079f492f83ab1582f8d46893',\n",
+       " 'fd4e785b8d9c4d7bb8af493c54ee6870',\n",
+       " 'd91c68bc847c4c5993e6fa53b657b504',\n",
+       " '1bfd2c9fa301401a91ff49843b1c842c',\n",
+       " '9684447b6a9044339ff355877d86f7d3',\n",
+       " '6fcdfdbf98c64c8b89a3730f72f8268f',\n",
+       " '6ccdf85fa1ce4bd19775b7fcf5a12ee7',\n",
+       " 'e67bb1231dce4f35b3914e3a40bb9c12',\n",
+       " '7a55571c1f654084844ec308bba0ba42',\n",
+       " '38d650edd70742898f976233a7dfb85a',\n",
+       " '496b340000dc4eb09780ce18b3ba5392',\n",
+       " 'a79b3b9c6ea74599885fdb8d28d12cfd',\n",
+       " 'f4de9dfe9b9c47f7aee86802c145d2d4',\n",
+       " 'c5b4d1fed5874094acaadc869998174f',\n",
+       " 'd4ca3a34bfeb41bc8065e682213eaaab',\n",
+       " '40786361d73e4a58a173a099821b3020',\n",
+       " 'c267c513aa0049168f2ab2e2444029c9',\n",
+       " 'de47fda5d45340b58a4febf243c18c90',\n",
+       " 'b8f6f44e9ca64d28956c159f9aa284bb',\n",
+       " 'cb6cd0c5ffb743ce8d07d0c02ed2cbe3',\n",
+       " 'f42720e5bcd94c4ab0f3880cf75dbb50',\n",
+       " '3bd78527eff54c5db6ead2f0471d1b55',\n",
+       " 'c5257e382fbc4f69a16aa0bc047dfee2',\n",
+       " '22df791dc8fa45c9867e2cd4de171bd9',\n",
+       " 'b3a0c0feba764bd0abfb446204a8239f',\n",
+       " 'ba39d1b49cf840289c2d2d04e88948cd',\n",
+       " '1a956280d4db49aea6007c9c1d0f698a',\n",
+       " 'e7062f6c5dba476facf895b6faee99cd',\n",
+       " '95f2febaef5a433f89c11d3e9741347f',\n",
+       " 'fded3452cdbf42fa90f7fadfacd5dd63',\n",
+       " '0a89bd45fc9d4148828cddb02a0921e7']"
+      ]
+     },
+     "execution_count": 25,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# Vector Store with recursive chunked documents\n",
+    "\n",
+    "recursive_collection_name = \"ai-safety-sr-arctic-embed-l-recursive\"\n",
+    "\n",
+    "recursive_qdrant_client = QdrantClient(url=qdrant_server,api_key=os.environ[\"QDRANT_API_KEY\"])\n",
+    "# recursive_qdrant_client.create_collection(\n",
+    "#     collection_name=recursive_collection_name,\n",
+    "#     vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),\n",
+    "# )\n",
+    "\n",
+    "recursive_vector_store = QdrantVectorStore(\n",
+    "    client=recursive_qdrant_client,\n",
+    "    collection_name=recursive_collection_name,\n",
+    "    embedding=embedding_model,\n",
+    ")\n",
+    "\n",
+    "recursive_vector_store.add_documents(recursive_chunked_docs)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "recursive_retriever = recursive_vector_store.as_retriever(search_type=\"similarity_score_threshold\",\n",
+    "                search_kwargs={'k':10,'score_threshold': 0.8})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 11, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': '70fb8aa0-96a7-4d0a-9757-05ac44f08577', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='FROM \\nPRINCIPLES \\nTO PRACTICE \\nA TECHINCAL COMPANION TO\\nTHE Blueprint for an \\nAI BILL OF RIGHTS\\n12'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 50, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': 'e254c679-f27c-4143-bcdd-a32f15e846e1', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='• Accessibility and reasonable \\naccommodations \\n• AI actor credentials and qualifications  \\n• Alignment to organizational values \\n• Auditing and assessment \\n• Change-management controls \\n• Commercial use \\n• Data provenance'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 19, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': '2c07a476-9e85-425a-9a32-a053f1293ff2', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='organization’s business processes or other activities, system goals, any human-run procedures that form a \\npart of the system, and specific performance expectations; a description of any data used to train machine \\nlearning models or for other purposes, including how data sources were processed and interpreted, a \\nsummary of what data might be missing, incomplete, or erroneous, and data relevancy justifications; the \\nresults of public consultation such as concerns raised and any decisions made due to these concerns; risk \\nidentification and management assessments and any steps taken to mitigate potential harms; the results of \\nperformance testing including, but not limited to, accuracy, differential demographic impact, resulting \\nerror rates (overall and per demographic group), and comparisons to previously deployed systems; \\nongoing monitoring procedures and regular performance testing reports, including monitoring frequency,'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 51, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': 'e08419ed-7e15-4a7d-a627-45b3bd5ebd78', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='lifecycle and informed by representative AI Actors (see Figure 3 of the AI RMF). Until new and rigorous'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 25, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': 'd9ebc600-4412-4f0a-890f-7836cb58f4a2', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='for any resulting algorithmic discrimination.   \\n26\\nAlgorithmic \\nDiscrimination \\nProtections'),\n",
+       " Document(metadata={'source': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'file_path': 'https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf', 'page': 0, 'total_pages': 73, 'format': 'PDF 1.6', 'title': 'Blueprint for an AI Bill of Rights', 'author': '', 'subject': '', 'keywords': '', 'creator': 'Adobe Illustrator 26.3 (Macintosh)', 'producer': 'iLovePDF', 'creationDate': \"D:20220920133035-04'00'\", 'modDate': \"D:20221003104118-04'00'\", 'trapped': '', '_id': 'dd370438-231c-41db-b7b1-b4f1e7673cf7', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='BLUEPRINT FOR AN \\nAI BILL OF \\nRIGHTS \\nMAKING AUTOMATED \\nSYSTEMS WORK FOR \\nTHE AMERICAN PEOPLE \\nOCTOBER 2022'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 38, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '7b595edf-6178-4a54-9e64-edfc1e18a497', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='guide the design of provenance data-tracking techniques. \\nHuman-AI Configuration; \\nInformation Integrity \\nMS-2.10-003 Verify deduplication of GAI training data samples, particularly regarding synthetic \\ndata. \\nHarmful Bias and Homogenization \\nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, End-Users, Operation and Monitoring, TEVV'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 59, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': 'b8f6f44e-9ca6-4d28-956c-159f9aa284bb', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='https://www.bloomberg.com/graphics/2023-generative-ai-bias/. \\nNational Institute of Standards and Technology (2024) Adversarial Machine Learning: A Taxonomy and \\nTerminology of Attacks and Mitigations https://csrc.nist.gov/pubs/ai/100/2/e2023/final \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework. \\nhttps://www.nist.gov/itl/ai-risk-management-framework \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 3: AI \\nRisks and Trustworthiness. \\nhttps://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Foundational_Information/3-sec-characteristics \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Chapter 6: AI \\nRMF Profiles. https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF/Core_And_Profiles/6-sec-profile \\nNational Institute of Standards and Technology (2023) AI Risk Management Framework, Appendix A: \\nDescriptions of AI Actor Tasks.'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 57, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': '38d650ed-d707-4289-8f97-6233a7dfb85a', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='54 \\nAppendix B. References \\nAcemoglu, D. (2024) The Simple Macroeconomics of AI https://www.nber.org/papers/w32487 \\nAI Incident Database. https://incidentdatabase.ai/ \\nAtherton, D. (2024) Deepfakes and Child Safety: A Survey and Analysis of 2023 Incidents and Responses. \\nAI Incident Database. https://incidentdatabase.ai/blog/deepfakes-and-child-safety/ \\nBadyal, N. et al. (2023) Intentional Biases in LLM Responses. arXiv. https://arxiv.org/pdf/2311.07611 \\nBing Chat: Data Exfiltration Exploit Explained. Embrace The Red. \\nhttps://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/ \\nBommasani, R. et al. (2022) Picking on the Same Person: Does Algorithmic Monoculture lead to Outcome \\nHomogenization? arXiv. https://arxiv.org/pdf/2211.13972 \\nBoyarskaya, M. et al. (2020) Overcoming Failures of Imagination in AI Infused System Development and \\nDeployment. arXiv. https://arxiv.org/pdf/2011.13416 \\nBrowne, D. et al. (2023) Securing the AI Pipeline. Mandiant.'),\n",
+       " Document(metadata={'source': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'file_path': 'https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf', 'page': 12, 'total_pages': 64, 'format': 'PDF 1.6', 'title': 'Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile', 'author': 'National Institute of Standards and Technology', 'subject': '', 'keywords': '', 'creator': 'Acrobat PDFMaker 24 for Word', 'producer': 'Adobe PDF Library 24.2.159', 'creationDate': \"D:20240805141702-04'00'\", 'modDate': \"D:20240805143048-04'00'\", 'trapped': '', '_id': 'd5048b39-7eb0-4c6f-b97a-083b66aa6ac2', '_collection_name': 'ai-safety-sr-arctic-embed-l-recursive'}, page_content='Priorities Related to Information Integrity Research and Development.')]"
+      ]
+     },
+     "execution_count": 28,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "recursive_retriever.invoke(\"What steps can organizations take to minimize bias in AI models?\")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Trying Compression retriver\n",
+    "from langchain.retrievers import ContextualCompressionRetriever\n",
+    "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
+    "from langchain_openai import ChatOpenAI\n",
+    "\n",
+    "base_retriever = recursive_retriever\n",
+    "\n",
+    "#Create a contextual compressor\n",
+    "compressor_llm = ChatOpenAI(temperature=0, model_name=\"gpt-4o\", max_tokens=4000)\n",
+    "compressor = LLMChainExtractor.from_llm(compressor_llm)\n",
+    "\n",
+    "#Combine the retriever with the compressor\n",
+    "compression_retriever = ContextualCompressionRetriever(\n",
+    "    base_compressor=compressor,\n",
+    "    base_retriever=base_retriever\n",
+    ")\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"Enter Your OpenAI API Key: \")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "\n",
+    "RAG_PROMPT = \"\"\"\\\n",
+    "Given a provided context and question, you must answer the question based only on context.\n",
+    "\n",
+    "If you cannot answer the question based on the context - you must say \"I don't know\".\n",
+    "\n",
+    "Context: {context}\n",
+    "Question: {question}\n",
+    "\"\"\"\n",
+    "\n",
+    "rag_prompt = ChatPromptTemplate.from_template(RAG_PROMPT)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_openai import ChatOpenAI\n",
+    "\n",
+    "# Using the same model used in the app.\n",
+    "chat_model_name = \"gpt-4o\"\n",
+    "llm = ChatOpenAI(model=chat_model_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 38,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from operator import itemgetter\n",
+    "from langchain_core.runnables import RunnablePassthrough, RunnableParallel\n",
+    "from langchain.schema import StrOutputParser\n",
+    "\n",
+    "ai_safety_rag_chain = (\n",
+    "    {\"context\": itemgetter(\"question\") | compression_retriever, \"question\": itemgetter(\"question\")}\n",
+    "    | rag_prompt | llm | StrOutputParser()\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "'Companies can ensure AI does not violate data privacy laws by incorporating built-in protections and ensuring that data collection conforms to reasonable expectations. They should collect only the data strictly necessary for the specific context and seek user permission, respecting their decisions regarding the collection, use, access, transfer, and deletion of data. If obtaining user permission is not possible, alternative privacy by design safeguards should be used. Additionally, systems should avoid user experience and design decisions that obfuscate user choice or burden users with privacy-invasive defaults.'"
+      ]
+     },
+     "execution_count": 39,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ai_safety_rag_chain.invoke({\"question\" : \"How can companies ensure AI does not violate data privacy laws?\"})"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "\"I don't know.\""
+      ]
+     },
+     "execution_count": 40,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "ai_safety_rag_chain.invoke({\"question\" : \"What are the implications of using GAI systems for organizations in terms of risk management and compliance?\"})"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

Tasks/Task 2/Task2.md

@@ -0,0 +1,58 @@
+# Deliverable 1
+Build a prototype and deploy to a Hugging Face Space, and create a short (< 2 min) loom video demonstrating some initial testing inputs and outputs.
+
+
+# Deliverable 2
+How did you choose your stack, and why did you select each tool the way you did?
+
+In choosing my stack, I aimed to balance performance, scalability, and the ability to handle both structured and unstructured data. The tools I selected provide flexibility to handle diverse document types, while ensuring effective chunking and retrieval. Here's the breakdown of each tool and why it was chosen:
+
+Qdrant ???
+## 1. **PyMuPDFLoader**:
+   - **Reason for Selection**: 
+     - PyMuPDFLoader is fast, lightweight, and efficient for parsing PDFs. It offers good performance with respect to speed and memory usage, which is crucial when dealing with large documents.
+     - In my use case, the AI Bill of Rights and NIST RMF documents are both structured and relatively dense. PyMuPDFLoader allows for quick loading and extraction of content without compromising on accuracy.
+   - **Why Not Another Option**: 
+     - While I considered `PyPDFium2Loader`, it is slower (2 minutes 30 seconds for loading the same document) and the performance difference didn’t justify the switch since the output quality between the two loaders was almost identical.
+
+## 2. **RecursiveCharacterTextSplitter**:
+   - **Reason for Selection**: 
+     - This splitter allows for flexible chunking of documents into manageable pieces while preserving context and meaning. It is particularly effective because it prevents breaking up the text in the middle of a thought, ensuring that the chunks remain semantically coherent.
+     - It also enables me to adjust the chunk sizes dynamically based on the document's structure. For instance, with the Blueprint for AI Bill of Rights, I can chunk based on sections, principles, and subsections, while still applying the RecursiveCharacter strategy within each chunk.
+   - **Why Not a Static Chunking Strategy**: 
+     - A simple page or sentence-based chunking would not retain the full context in many cases. The recursive strategy ensures that chunks are more comprehensive, making retrieval more effective.
+
+## 3. **SemanticChunker**:
+   - **Reason for Selection**: 
+     - This chunker allows for semantically rich divisions of the text, meaning that chunks are more likely to contain entire ideas or thoughts. This approach enhances coherence and leads to better retrieval outcomes.
+     - It is also adaptable and can be used to refine chunking strategies for documents that may not be as well-structured as the AI Bill of Rights or NIST RMF documents.
+   - **Why This Over Simple Chunking**: 
+     - Semantic chunking provides better retrieval precision, especially in answering complex questions, since the context is more meaningful. This is particularly important when documents do not follow a clear structure.
+
+## 4. **Snowflake-Arctic-Embed-L Embedding Model**:
+   - **Reason for Selection**: 
+     - This model offers a good balance between performance and accuracy. With 334 million parameters and 1024-dimension embeddings, it is a smaller model but ranks competitively on the MTEB leaderboard (27th), suggesting its efficiency.
+     - For a retrieval-augmented generation (RAG) setup, the embedding model plays a critical role in vectorizing chunks accurately, and this model is performant for both speed and relevance in retrieval tasks.
+   - **Why Not a Larger Model**: 
+     - Larger models with more parameters may improve accuracy slightly but come at a much higher computational cost. For enterprise applications, the smaller yet efficient `snowflake-arctic-embed-l` model provides a good trade-off between speed and accuracy, allowing for scalability without major infrastructure demands.
+
+## 5. **Context Enrichment and Contextual Compression**:
+   - **Reason for Selection**: 
+     - These advanced retrieval techniques aim to enhance the quality of responses by improving the retrieval process. Context enrichment allows for richer, more informed responses, while contextual compression ensures that responses remain concise and relevant.
+   - **Why Not Pure Contextual Retrieval**: 
+     - Pure retrieval may lead to irrelevant or verbose results. By applying these techniques, I ensure that the retrieval process generates more targeted and meaningful answers, which is essential when dealing with complex or nuanced questions (e.g., AI ethics, privacy, and risk management).
+
+## 6. **Grouping by Similar Context**:
+   - **Reason for Selection**: 
+     - Grouping documents by similar context improves retrieval accuracy. When a user asks about a specific topic like data privacy, the system can retrieve relevant chunks from different documents (e.g., both the AI Bill of Rights and NIST RMF), ensuring that responses are comprehensive.
+   - **Why This Strategy**: 
+     - Grouping chunks by similar context ensures that even when documents are diverse or cover multiple topics, the right content is prioritized during retrieval. This helps improve answer quality, especially when dealing with detailed or nuanced questions.
+
+## 7. **Vector Store**:
+   - **Reason for Selection**: 
+     - Using a vector store enables efficient storage and retrieval of embeddings, ensuring fast lookups and scalable operations. It also allows for advanced similarity search, making sure the most relevant chunks are retrieved based on the query embeddings.
+   - **Why Not Traditional Indexing**: 
+     - Traditional indexing methods are less effective in handling semantic content and would not allow for the nuanced retrieval that RAG applications require. Vector stores enable better handling of embeddings and can scale with large datasets.
+
+## Conclusion:
+Each tool in this stack was chosen to ensure **speed**, **scalability**, and **accuracy** while dealing with structured and unstructured documents. By balancing performance with precision (e.g., fast document loading via PyMuPDFLoader, efficient chunking strategies, and a small but powerful embedding model), this stack provides a robust framework for building ethical and useful AI applications.

Tasks/Task 3/RAGAS_Key_Metrics_Summary.csv

@@ -0,0 +1,6 @@
+,Mean,Standard Deviation,25th Percentile,50th Percentile (Median),75th Percentile
+faithfulness,0.5825636951112327,0.40610877667716644,0.08083333333333333,0.7,1.0
+answer_relevancy,0.9421626112319366,0.1563145734421645,0.9458433953002494,0.974098728728228,0.9933414393143032
+context_recall,0.27157629475147727,0.3958207397121211,0.0,0.0,0.5
+context_precision,0.44603102186282584,0.48784278923685864,0.0,0.0,0.9999999999
+answer_correctness,0.6178986199147979,0.2533140961114008,0.38881797261860573,0.6183298094933991,0.8654874927504908

Tasks/Task 3/Task3.md

@@ -0,0 +1,38 @@
+
+# Deliverable 1
+
+Assess your pipeline using the RAGAS framework including key metrics faithfulness, answer relevancy, context precision, and context recall.  Provide a table of your output results.
+
+## Pipeline configuration
+ 1. Qdrant - Cloud hosted vector database
+ 2. PyMuPdfLoader - Pdf loader from langchain
+ 3. Snowflake/snowflake-arctic-embed-l - Open source embedding model used
+ 4. SemanticChunker & RecursiveCharacterText with contextual compression - Chunking strategy [Note that SemanticChunker seems to be unreliable and produces duplicate chunks]
+ 5. gpt-4o-mini - Generator LLM
+ 6. gpt-40 - Critic LLM
+ 7. Distribution - simple 0.5, multi_context 0.3 and reasoning 0.2
+ 8. Ragas metrics -   faithfulness, answer_relevancy, context_recall, context_precision,answer_correctness
+ 9. Sythetic Questions generated - 269
+
+![tak3-del1](/task3-del1.png)
+
+![tak3-del1](/task3-del11.png)
+
+# Deliverable 2
+What conclusions can you draw about performance and effectiveness of your pipeline with this information?
+
+
+## Observations:
+Some observations from the results
+
+- **Faithfulness**: Mostly high faithfulness scores, indicating that the generated answers are generally true to the source material. But, there are some low score (e.g., 0.233333) which shows that the model may occasionally provide unfaithful or incomplete answers.
+  
+- **Answer Relevancy**: The model seems to perform well in answer relevancy, with most scores being near 1. This suggests that even when faithfulness is low, the answers provided are still on-topic and relevant to the user's question.
+
+- **Context Recall & Precision**: There are several instances where **context recall** is 0.0, indicating that the context was not helpful in answering the question. However, when context recall is high, **context precision** is often perfect (1.0), showing that when the context is relevant, it is precise and accurate.
+
+- **Answer Correctness**: This metric shows a range of results. Although many answers are correct, a few are only partially correct, suggesting room for improvement in the correctness of generated answers.
+
+The pipeline performs well in generating relevant answers, but some improvements can be made to enhance the faithfulness and correctness of those answers.
+
+The **context recall** metric has room for improvement. There are several cases where relevant context is missing or inadequate, which can impact the overall effectiveness of the pipeline.

Tasks/Task 3/task3-generate-dataset-ragas-eval.ipynb

@@ -0,0 +1,590 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Synthetic data generation using Ragas framework"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Python packages are installed from `requirements.txt` file into virtual environment"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -qU langsmith langchain-core langchain-community langchain-openai langchain-qdrant langchain_experimental pymupdf ragas"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "aiofiles==23.2.1\n",
+      "aiohappyeyeballs==2.4.0\n",
+      "aiohttp==3.10.5\n",
+      "aiosignal==1.3.1\n",
+      "annotated-types==0.7.0\n",
+      "anyio==3.7.1\n",
+      "appdirs==1.4.4\n",
+      "appnope==0.1.4\n",
+      "asttokens==2.4.1\n",
+      "asyncer==0.0.2\n",
+      "attrs==24.2.0\n",
+      "bidict==0.23.1\n",
+      "certifi==2024.8.30\n",
+      "chainlit==0.7.700\n",
+      "charset-normalizer==3.3.2\n",
+      "click==8.1.7\n",
+      "comm==0.2.2\n",
+      "dataclasses-json==0.5.14\n",
+      "datasets==3.0.0\n",
+      "debugpy==1.8.5\n",
+      "decorator==5.1.1\n",
+      "Deprecated==1.2.14\n",
+      "dill==0.3.8\n",
+      "distro==1.9.0\n",
+      "executing==2.1.0\n",
+      "fastapi==0.100.1\n",
+      "fastapi-socketio==0.0.10\n",
+      "filelock==3.16.1\n",
+      "filetype==1.2.0\n",
+      "frozenlist==1.4.1\n",
+      "fsspec==2024.6.1\n",
+      "googleapis-common-protos==1.65.0\n",
+      "grpcio==1.66.1\n",
+      "grpcio-tools==1.62.3\n",
+      "h11==0.14.0\n",
+      "h2==4.1.0\n",
+      "hpack==4.0.0\n",
+      "httpcore==0.17.3\n",
+      "httpx==0.24.1\n",
+      "huggingface-hub==0.25.0\n",
+      "hyperframe==6.0.1\n",
+      "idna==3.10\n",
+      "importlib_metadata==8.4.0\n",
+      "ipykernel==6.29.5\n",
+      "ipython==8.27.0\n",
+      "jedi==0.19.1\n",
+      "Jinja2==3.1.4\n",
+      "jiter==0.5.0\n",
+      "joblib==1.4.2\n",
+      "jsonpatch==1.33\n",
+      "jsonpointer==3.0.0\n",
+      "jupyter_client==8.6.3\n",
+      "jupyter_core==5.7.2\n",
+      "langchain==0.3.0\n",
+      "langchain-community==0.3.0\n",
+      "langchain-core==0.3.5\n",
+      "langchain-experimental==0.3.0\n",
+      "langchain-huggingface==0.1.0\n",
+      "langchain-openai==0.2.0\n",
+      "langchain-qdrant==0.1.4\n",
+      "langchain-text-splitters==0.3.0\n",
+      "langsmith==0.1.125\n",
+      "Lazify==0.4.0\n",
+      "MarkupSafe==2.1.5\n",
+      "marshmallow==3.22.0\n",
+      "matplotlib-inline==0.1.7\n",
+      "mpmath==1.3.0\n",
+      "multidict==6.1.0\n",
+      "multiprocess==0.70.16\n",
+      "mypy-extensions==1.0.0\n",
+      "nest-asyncio==1.6.0\n",
+      "networkx==3.3\n",
+      "numpy==1.26.4\n",
+      "openai==1.44.1\n",
+      "opentelemetry-api==1.27.0\n",
+      "opentelemetry-exporter-otlp==1.27.0\n",
+      "opentelemetry-exporter-otlp-proto-common==1.27.0\n",
+      "opentelemetry-exporter-otlp-proto-grpc==1.27.0\n",
+      "opentelemetry-exporter-otlp-proto-http==1.27.0\n",
+      "opentelemetry-instrumentation==0.48b0\n",
+      "opentelemetry-proto==1.27.0\n",
+      "opentelemetry-sdk==1.27.0\n",
+      "opentelemetry-semantic-conventions==0.48b0\n",
+      "orjson==3.10.7\n",
+      "packaging==23.2\n",
+      "pandas==2.2.3\n",
+      "parso==0.8.4\n",
+      "pexpect==4.9.0\n",
+      "pillow==10.4.0\n",
+      "platformdirs==4.3.6\n",
+      "portalocker==2.10.1\n",
+      "prompt_toolkit==3.0.47\n",
+      "protobuf==4.25.5\n",
+      "psutil==6.0.0\n",
+      "ptyprocess==0.7.0\n",
+      "pure_eval==0.2.3\n",
+      "pyarrow==17.0.0\n",
+      "pydantic==2.9.2\n",
+      "pydantic-settings==2.5.2\n",
+      "pydantic_core==2.23.4\n",
+      "Pygments==2.18.0\n",
+      "PyJWT==2.9.0\n",
+      "PyMuPDF==1.24.10\n",
+      "pymupdf4llm==0.0.17\n",
+      "PyMuPDFb==1.24.10\n",
+      "pypdf==4.3.1\n",
+      "pysbd==0.3.4\n",
+      "python-dateutil==2.9.0.post0\n",
+      "python-dotenv==1.0.1\n",
+      "python-engineio==4.9.1\n",
+      "python-graphql-client==0.4.3\n",
+      "python-multipart==0.0.6\n",
+      "python-socketio==5.11.4\n",
+      "pytz==2024.2\n",
+      "PyYAML==6.0.2\n",
+      "pyzmq==26.2.0\n",
+      "qdrant-client==1.11.2\n",
+      "ragas==0.1.19\n",
+      "regex==2024.9.11\n",
+      "requests==2.32.3\n",
+      "safetensors==0.4.5\n",
+      "scikit-learn==1.5.2\n",
+      "scipy==1.14.1\n",
+      "sentence-transformers==3.1.1\n",
+      "simple-websocket==1.0.0\n",
+      "six==1.16.0\n",
+      "sniffio==1.3.1\n",
+      "SQLAlchemy==2.0.35\n",
+      "stack-data==0.6.3\n",
+      "starlette==0.27.0\n",
+      "sympy==1.13.3\n",
+      "syncer==2.0.3\n",
+      "tenacity==8.5.0\n",
+      "threadpoolctl==3.5.0\n",
+      "tiktoken==0.7.0\n",
+      "tokenizers==0.19.1\n",
+      "tomli==2.0.1\n",
+      "torch==2.4.1\n",
+      "tornado==6.4.1\n",
+      "tqdm==4.66.5\n",
+      "traitlets==5.14.3\n",
+      "transformers==4.44.2\n",
+      "typing-inspect==0.9.0\n",
+      "typing_extensions==4.12.2\n",
+      "tzdata==2024.1\n",
+      "uptrace==1.26.0\n",
+      "urllib3==2.2.3\n",
+      "uvicorn==0.23.2\n",
+      "watchfiles==0.20.0\n",
+      "wcwidth==0.2.13\n",
+      "websockets==13.1\n",
+      "wrapt==1.16.0\n",
+      "wsproto==1.2.0\n",
+      "xxhash==3.5.0\n",
+      "yarl==1.11.1\n",
+      "zipp==3.20.2\n"
+     ]
+    }
+   ],
+   "source": [
+    "!pip freeze\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "from uuid import uuid4\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass(\"LangChain API Key:\")\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_PROJECT\"] = \"AIM-SDG-MidTerm - AI Safety\"\n",
+    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
+    "\n",
+    "os.environ[\"QDRANT_API_KEY\"] = getpass.getpass(\"Enter Your Qdrant API Key: \")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from pdfloader import PDFLoaderWrapper\n",
+    "from langchain_experimental.text_splitter import SemanticChunker\n",
+    "\n",
+    "BOR_FILE_PATH = \"https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf\"\n",
+    "NIST_FILE_PATH = \"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf\"\n",
+    "SMALL_DOC = \"https://arxiv.org/pdf/1908.10084\"  \n",
+    "documents_to_preload = [\n",
+    "    BOR_FILE_PATH,\n",
+    "    NIST_FILE_PATH\n",
+    "    # SMALL_DOC\n",
+    "]\n",
+    "\n",
+    "pdf_loader = PDFLoaderWrapper(\n",
+    "    documents_to_preload, PDFLoaderWrapper.LoaderType.PYMUPDF\n",
+    ")\n",
+    "documents = await pdf_loader.aload()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "print (\"Importing packages\")\n",
+    "from ragas.testset.generator import TestsetGenerator\n",
+    "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
+    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
+    "from ragas.testset.docstore import Document, DocumentStore,InMemoryDocumentStore\n",
+    "from langchain_experimental.text_splitter import SemanticChunker\n",
+    "from langchain_huggingface import HuggingFaceEmbeddings, HuggingFacePipeline\n",
+    "from ragas.testset.extractor import KeyphraseExtractor\n",
+    "\n",
+    "print (\"Packages import complete\")\n",
+    "print (\"Getting the Embedding model from Huggingface\")\n",
+    "# Using best performing embedding model from hugging face to generate quality dataset.\n",
+    "# Need GPU\n",
+    "model_name = \"Snowflake/snowflake-arctic-embed-l\"\n",
+    "embedding_model = HuggingFaceEmbeddings(model_name=model_name)\n",
+    "print (\"Embedding model loaded\")\n",
+    "\n",
+    "print (\"Splitting the documents into semantic chunks\")\n",
+    "text_splitter = SemanticChunker(embedding_model, breakpoint_threshold_type=\"percentile\",breakpoint_threshold_amount=90)\n",
+    "chunked_docs = text_splitter.split_documents(documents)\n",
+    "\n",
+    "print (\"Creating the document store for ragas and loading LLM models\")\n",
+    "generator_llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
+    "critic_llm = ChatOpenAI(model=\"gpt-4o\")\n",
+    "\n",
+    "keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)\n",
+    "docstore = InMemoryDocumentStore(splitter=text_splitter,extractor=keyphrase_extractor, embeddings=embedding_model)\n",
+    "\n",
+    "\n",
+    "print (\"Creating the testset generator\")\n",
+    "generator = TestsetGenerator.from_langchain( # Default uses TokenTextSplitter\n",
+    "    generator_llm=generator_llm,\n",
+    "    critic_llm=critic_llm,\n",
+    "    embeddings=embedding_model,\n",
+    "    docstore=docstore # Document store uses SemenaticChunker\n",
+    ")\n",
+    "\n",
+    "distributions = {\n",
+    "    simple: 0.5,\n",
+    "    multi_context: 0.3,\n",
+    "    reasoning: 0.2\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "tests_per_doc = 2 \n",
+    "test_size = tests_per_doc * len(documents)\n",
+    "\n",
+    "testset = generator.generate_with_langchain_docs(\n",
+    "    documents, \n",
+    "    test_size, \n",
+    "    distributions, \n",
+    "    with_debugging_logs=True\n",
+    ") # Default  RunConfig(max_retries=15, max_wait=90)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "testset.to_pandas()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langsmith import Client\n",
+    "\n",
+    "client = Client()\n",
+    "\n",
+    "dataset_name = \"AI Safety\"\n",
+    "\n",
+    "dataset = client.create_dataset(\n",
+    "    dataset_name=dataset_name,\n",
+    "    description=\"Questions about AI Safety\"\n",
+    ")\n",
+    "\n",
+    "for test in testset.to_pandas().iterrows():\n",
+    "  client.create_example(\n",
+    "      inputs={\n",
+    "          \"question\": test[1][\"question\"]\n",
+    "      },\n",
+    "      outputs={\n",
+    "          \"answer\": test[1][\"ground_truth\"]\n",
+    "      },\n",
+    "      metadata={\n",
+    "          \"context\": test[0]\n",
+    "      },\n",
+    "      dataset_id=dataset.id\n",
+    "  )"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create Rag chain to generate answers for above questions in the dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Note that we are usig Qdrant cloud where the pdf document is processed and saved for us to consume. For the RAG pipeline we use the same embedding model originally used to populate the Qdrant vectorstore."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_qdrant import QdrantVectorStore\n",
+    "from langchain_core.documents import Document\n",
+    "from qdrant_client import QdrantClient\n",
+    "from qdrant_client.http.models import Distance, VectorParams\n",
+    "\n",
+    "dimension = 1024\n",
+    "collection_name = \"ai-safety-sr-arctic-embed-l-semantic\"\n",
+    "qdrant_server = \"https://500cb0e8-ea08-4662-b4f2-3eca11e635da.europe-west3-0.gcp.cloud.qdrant.io:6333\"\n",
+    "qdrant_client = QdrantClient(url=qdrant_server,api_key=os.environ[\"QDRANT_API_KEY\"])\n",
+    "qdrant_client.create_collection(\n",
+    "    collection_name=collection_name,\n",
+    "    vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),\n",
+    ")\n",
+    "\n",
+    "vector_store = QdrantVectorStore(\n",
+    "    client=qdrant_client,\n",
+    "    collection_name=collection_name,\n",
+    "    embedding=embedding_model,\n",
+    ")\n",
+    "\n",
+    "retriever = vector_store.as_retriever()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "\n",
+    "RAG_PROMPT = \"\"\"\\\n",
+    "Given a provided context and question, you must answer the question based only on context.\n",
+    "\n",
+    "If you cannot answer the question based on the context - you must say \"I don't know\".\n",
+    "\n",
+    "Context: {context}\n",
+    "Question: {question}\n",
+    "\"\"\"\n",
+    "\n",
+    "rag_prompt = ChatPromptTemplate.from_template(RAG_PROMPT)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_openai import ChatOpenAI\n",
+    "\n",
+    "# Using the same model used in the app.\n",
+    "chat_model_name = \"gpt-4o\"\n",
+    "llm = ChatOpenAI(model=chat_model_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from operator import itemgetter\n",
+    "from langchain_core.runnables import RunnablePassthrough, RunnableParallel\n",
+    "from langchain.schema import StrOutputParser\n",
+    "\n",
+    "ai_safety_rag_chain = (\n",
+    "    {\"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\")}\n",
+    "    | rag_prompt | llm | StrOutputParser()\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "ai_safety_rag_chain.invoke({\"question\" : \"What steps can organizations take to minimize bias in AI models?\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# LangSmith Evaluation setup"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langsmith.evaluation import LangChainStringEvaluator, evaluate\n",
+    "\n",
+    "eval_llm = ChatOpenAI(model=\"gpt-4o\")\n",
+    "\n",
+    "# Evaluators\n",
+    "qa_evaluator = LangChainStringEvaluator(\"qa\", config={\"llm\" : eval_llm})\n",
+    "\n",
+    "# Faithfulness Evaluator\n",
+    "# Checks whether the generated answer is faithful to the provided source material or context.\n",
+    "faithfulness_evaluator = LangChainStringEvaluator(\n",
+    "    \"criteria\",\n",
+    "    config={\n",
+    "        \"criteria\": {\n",
+    "            \"faithfulness\": (\n",
+    "                \"Is the answer faithful to the given context?\"\n",
+    "            )\n",
+    "        },\n",
+    "        \"llm\": eval_llm\n",
+    "    },\n",
+    "    prepare_data=lambda run, example: {\n",
+    "        \"prediction\": run.outputs[\"output\"],\n",
+    "        \"reference\": example.outputs[\"answer\"],\n",
+    "        \"input\": example.inputs[\"question\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# Answer Relevancy Evaluator\n",
+    "# Determines whether the answer is relevant to the user's question.\n",
+    "answer_relevancy_evaluator = LangChainStringEvaluator(\n",
+    "    \"criteria\",\n",
+    "    config={\n",
+    "        \"criteria\": {\n",
+    "            \"relevancy\": (\n",
+    "                \"Does the answer address the question and provide relevant information?\"\n",
+    "            )\n",
+    "        },\n",
+    "        \"llm\": eval_llm\n",
+    "    },\n",
+    "    prepare_data=lambda run, example: {\n",
+    "        \"prediction\": run.outputs[\"output\"],\n",
+    "        \"reference\": example.outputs[\"answer\"],\n",
+    "        \"input\": example.inputs[\"question\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# Context Precision Evaluator\n",
+    "# Evaluates how precisely the answer uses information from the given context.\n",
+    "context_precision_evaluator = LangChainStringEvaluator(\n",
+    "    \"criteria\",\n",
+    "    config={\n",
+    "        \"criteria\": {\n",
+    "            \"context_precision\": (\n",
+    "                \"Does the answer precisely use information from the provided context?\"\n",
+    "            )\n",
+    "        },\n",
+    "        \"llm\": eval_llm\n",
+    "    },\n",
+    "    prepare_data=lambda run, example: {\n",
+    "        \"prediction\": run.outputs[\"output\"],\n",
+    "        \"reference\": example.outputs[\"answer\"],\n",
+    "        \"input\": example.inputs[\"question\"],\n",
+    "    }\n",
+    ")\n",
+    "\n",
+    "# Context Recall Evaluator\n",
+    "# Determines if the answer recalls all the necessary and relevant information from the context.\n",
+    "context_recall_evaluator = LangChainStringEvaluator(\n",
+    "    \"criteria\",\n",
+    "    config={\n",
+    "        \"criteria\": {\n",
+    "            \"context_recall\": (\n",
+    "                \"Does the answer recall all relevant information from the provided context?\"\n",
+    "            )\n",
+    "        },\n",
+    "        \"llm\": eval_llm\n",
+    "    },\n",
+    "    prepare_data=lambda run, example: {\n",
+    "        \"prediction\": run.outputs[\"output\"],\n",
+    "        \"reference\": example.outputs[\"answer\"],\n",
+    "        \"input\": example.inputs[\"question\"],\n",
+    "    }\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "evaluate(\n",
+    "    ai_safety_rag_chain.invoke,\n",
+    "    data=dataset_name,\n",
+    "    evaluators=[\n",
+    "        qa_evaluator,\n",
+    "        faithfulness_evaluator,\n",
+    "        answer_relevancy_evaluator,\n",
+    "        context_precision_evaluator,\n",
+    "        context_recall_evaluator\n",
+    "    ],\n",
+    "    metadata={\"revision_id\": \"ai_safety_rag_chain\"},\n",
+    ")"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

Tasks/Task 4/test_dataset (2).jsonl

@@ -0,0 +1 @@
+{"questions": {"a188ff31-68db-46bc-b419-ac0fdecc6b1b": "What is data poisoning in the context of GAI and how does it affect model outputs?", "c9f79a82-8775-4461-ac2f-b7b4e8734e96": "How do GAI systems pose intellectual property risks related to copyrighted material in their training data?", "80b370f2-6d21-4ff6-bc9b-763317af5321": "How can governance structures help mitigate bias in human-based systems?", "d4e20011-f4f9-4de6-8453-8144d77ba3a7": "What are some common concerns associated with human-based systems in terms of bias and effectiveness?", "12534c5f-eb68-41cb-a879-b908e95a65d0": "What are the best practices for establishing policies for data collection and retention to mitigate risks such as data privacy breaches and harmful biases?", "11951856-b35d-4123-ab15-80c38cd82195": "How can organizations effectively implement policies to protect third-party intellectual property and training data?", "869c7e52-2b77-4144-af5f-d5beebc138f9": "What are the key safeguards that should be included in automated systems to protect the public from harm?", "a310a614-4bd4-4302-ab42-37b6faa7d838": "How can early-stage public consultation improve the safety and effectiveness of automated systems?", "468bff4b-cdbb-489a-9cbd-95d9062f93dc": "What are some of the key investigative projects that Surya Mattu has worked on at The Markup?", "3e5b4545-4ceb-424f-b519-16c3870dd543": "How has Mariah Montgomery's role as National Campaign Director at the Partnership for Working Families impacted labor rights and policies?", "0f0eea1d-f05e-458e-b31a-e955629c7e44": "How can organizations effectively integrate pre- and post-deployment feedback into their monitoring processes for GAI models?", "6cc0bcc5-9c82-49a6-b118-3e333e70ee9e": "What are the benefits of using AI red-teaming in the pre-deployment testing phase for capturing external feedback?", "4441faa1-8f27-4fc7-bea0-847caa1c1505": "What are the potential negative impacts of automated systems on individuals and communities?", "0db8fdee-99cf-47c6-9d8d-a85f3b294826": "How can confirmation bias affect the effectiveness of safety mechanisms in technology?", "57db460e-0123-4edf-b7df-87a967a60c26": "What are the key safety metrics used to evaluate AI system reliability and robustness?", "48589831-4f3c-4bf6-9cb4-bc4277c489dd": "How can AI systems be designed to fail safely when operating beyond their knowledge limits?", "1df11168-7aa5-4b43-91df-c14c32f01440": "What are the risks associated with data brokers collecting consumer data without permission?", "2127b35f-68cd-4e5f-a669-a6a4bb532fa8": "How does the use of surveillance technologies in schools and workplaces impact mental health?", "afefb290-48ec-450c-b530-5fe1b6c5340b": "What is ballot curing and how does it impact the election process?", "eecbf085-2f16-45c4-ba65-35813ca84568": "How do different states handle signature discrepancies in mail-in ballots?", "43b6add5-244e-4c11-be3b-0944fecfa6b9": "What are the best practices for detecting and mitigating algorithmic bias according to the Brookings Report?", "37bbd6b6-d24a-4b73-a4f4-f532d8c1793a": "How can public agencies implement Algorithmic Impact Assessments to ensure accountability, as suggested by the AI Now Institute Report?", "318fe73a-0591-41e8-b65e-925c71b2caab": "How is the federal government addressing discrimination in mortgage lending through the Department of Justice's nationwide initiative?", "56664bc2-0933-4e58-8d03-5c06b9d06c04": "What role do federal agencies like the Consumer Financial Protection Bureau play in the Action Plan to Advance Property Appraisal and Valuation Equity?", "7f8b418c-6e85-4ab0-83db-b7ed7dc49a45": "What are the best practices for updating due diligence processes to include intellectual property and data privacy for GAI acquisitions?", "e81617a3-9609-4012-ba46-caa374c306de": "How can organizations effectively monitor and assess third-party GAI risks in real-time?", "054e5797-d024-41bd-8de9-983d038a8797": "What are the best practices for performing disparity testing and making the results public?", "fdb81ad2-acf2-4aa4-b551-fe766d22f273": "How can organizations effectively mitigate disparities identified through testing?", "09a4ef32-a01e-4ca9-9bf6-4704e328ccef": "How can people protect themselves from being tracked by devices originally meant for finding lost items?", "e314f460-f6e2-4d11-b612-d51529a9dee6": "What are the potential issues with using algorithms to deploy police in neighborhoods?", "741f5989-422f-4bc5-9f72-0f3b22bb4f25": "What are the mental health impacts of NCII on women and sexual minorities?", "19592c9a-0621-4629-bdfc-8a08f0d396b4": "How can GAI training datasets be protected from including CSAM and NCII?", "f95100da-f55f-4402-909d-fdde5cf17d25": "What are the key challenges in designing non-discriminatory AI technology discussed in the panel?", "bb3e7970-5b1e-4e98-87ad-b30d33ff6a89": "How can community participation enhance human-computer interaction in AI systems?", "796ffa10-1532-4fa1-b832-d8ee058d410d": "What are the potential sociotechnical harms of algorithmic systems as discussed by Shelby et al (2023)?", "2c38117e-4b2d-4553-b319-f4ba3997996e": "How does training on generated data affect AI models according to Shumailov et al (2023)?", "3b9c9379-cc75-4b9d-a68a-dc6b0a48fd9c": "What are the key suggested actions for managing GAI risks according to the AI RMF 10 and Playbook?", "1a02235f-7bf0-4e7e-8149-ab610eacb769": "How do the suggested actions for managing GAI risks vary depending on the stage of the GAI lifecycle?", "08cbf993-d60b-4982-bf84-140c29d30450": "How can organizations ensure that consent practices do not allow for abusive surveillance practices?", "e253b5ac-feb7-4116-9e68-d2c817da36a5": "What are the best practices for re-acquiring consent if the use case of data changes or if data is transferred to another entity?", "0260750e-4f7d-4c1b-b0b7-ae4c36cc8fc3": "What are the key principles outlined in the AI Bill of Rights?", "39863570-2d41-4d21-bde1-1afc78c157b0": "How does the AI Bill of Rights address algorithmic discrimination?", "8b4fd9d7-e1d4-472e-bd34-35fa98299c07": "How can we effectively track and document instances of anthropomorphization in GAI system interfaces?", "1f34befe-4432-419f-8465-066a0d82ff77": "What are the best practices for verifying the provenance of GAI system training data and TEVV data?", "3e35db8c-c1b3-4b9f-b6c0-a3fd6e52d2b0": "What is the importance of having demographically and interdisciplinarily diverse AI red teams in pre-deployment contexts?", "0c3b35ca-f421-41e4-b016-8f367561acbe": "How can general public involvement in AI red-teaming contribute to identifying flaws in AI models?", "d73655e4-93f0-41c5-b69e-814ff8189db8": "QUESTION #1: How are major universities using race as a predictor of student success?", "bb8b9729-d1b6-407d-9f0c-aa1bd62a8d78": "QUESTION #2: What concerns do students and professors have about using race as a predictor in education?", "3f1dec42-4087-4e06-9e7e-491c96cdee67": "How can AI-enabled systems contribute to building better and more innovative infrastructure?", "9a94cfb2-25b9-4aa8-94c5-987c53fa42bf": "What lessons from urban planning can be applied to the integration of AI technologies in communities?", "f445449e-b75f-44ee-a819-018ad630bd35": "What are the benefits of having a human alternative to automated systems?", "58fd202f-0791-411a-9124-09381dbbad11": "How can one ensure timely human consideration and remedy when an automated system fails?", "26ee0f55-a947-440f-b4bc-4b7def4e3545": "What are the main findings of the Department of Justice's report on the risk assessment tool for predicting recidivism?", "dcb01564-a34f-42a7-ac6c-13764525a7d2": "How is the Department of Justice addressing the disparities in the risk assessment tool for predicting recidivism among different groups of color?", "8e29d29a-fc98-4a6f-b42b-580fc084dd71": "What is the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government?", "4bbc6d4b-6b67-4831-8bca-853eb46aec3a": "What were President Biden's remarks on the Supreme Court decision to overturn Roe v Wade?", "919fdd1d-2abb-472e-ac8d-bde9df2bb391": "What are the best practices for re-assessing model risks after implementing fine-tuning or retrieval-augmented generation?", "a795e873-419b-454b-8598-fb0c49a7e5cc": "How can organizations effectively review and manage training data to prevent the reproduction of intellectual property or CBRN information in AI outputs?", "242b750e-1236-41f7-a1cc-eedef8f0427d": "What are some common examples of AI incidents that organizations should be aware of?", "c506e557-776f-42ed-99f9-c752ac2bb94b": "How can organizations effectively track and document the provenance of datasets to identify AI-generated data issues?", "426616e2-6297-47c3-89c7-71ec1186cdba": "What is the role of the American Civil Liberties Union in protecting digital privacy?", "db43af55-434d-441f-8dc7-acc8ff3f8432": "How does the Center for Democracy & Technology advocate for internet freedom and security?", "f2913868-28a6-4558-904a-0486fbfc1f6e": "How can organizations ensure the accuracy of predictions or recommendations generated by automated systems?", "5a4faa70-0364-4fd0-9c98-b26fb63f7786": "What are the best practices for implementing ongoing monitoring procedures for automated systems?", "3ad57490-e4f7-4fd2-bff4-93211043ec13": "What are the key considerations for implementing automated systems in sensitive domains like criminal justice and health?", "3a05d7ba-2e46-406b-aeb9-51b33efff15f": "How can organizations ensure meaningful oversight and human consideration in high-risk automated decision-making systems?", "6ae09ea8-3090-401b-9f1e-4ce5270152cd": "What are the privacy expectations for automated systems handling sensitive data?", "207207ff-faab-4342-b76f-ef0c6fac88c9": "How should consent be managed for automated systems collecting sensitive data?", "d15a10aa-36cb-4f3a-9f9e-2c0416ce1084": "What is the contact information for inquiries related to NIST AI publications?", "c9f4fb11-9365-4354-aa94-7cc93efcafb5": "Where can I find additional information about NIST AI publications?", "4aebac20-11d4-42c8-be6a-f7ac4e43cbbc": "How can organizations effectively combat automation bias in automated systems?", "71537f88-7e77-4720-9cc0-bca516b4721f": "What are the best practices for training individuals to properly interpret outputs from automated systems?", "e004e796-65d5-4109-89bd-472cae5b6c75": "What were the FTC's findings in the case against Everalbum Inc?", "290cd0b2-456b-41bc-bf0e-3ea3e32f480d": "How did the FTC address privacy concerns in the case against Weight Watchers and Kurbo?", "1c416614-5e28-45f4-9e8e-937971dcff9a": "What are the potential harms of GAI related to misinformation, disinformation, and deepfakes?", "d83ab93d-9be0-488d-94fd-8e58074a3388": "How should organizations disclose the use of GAI to end users to mitigate risks?", "bfc45e93-d073-4348-8fb1-03dfaf4e73f3": "What measures can designers and developers take to prevent algorithmic discrimination?", "4819bdb4-1724-4318-855c-9c4f680c0655": "How does algorithmic discrimination impact different protected classes such as race, gender, and disability?", "a8a96840-d387-42d9-9b56-f05b73027f5c": "What are some innovative solutions provided by the industry to mitigate risks to the safety and efficacy of AI systems?", "7fdb6c15-c3f8-4327-b2fe-0169c08ce375": "How does the Office of Management and Budget (OMB) suggest expanding opportunities for stakeholder engagement in program design?", "3509c40f-7af0-49a5-bd16-c7da584b3980": "What are the nine principles outlined in Executive Order 13960 for the use of AI in the federal government?", "a86eba64-72a8-4afa-a7f5-8c50c3b0c660": "How can laws and policies ensure that AI systems are accurate, reliable, and effective in real-life applications?", "9eee9d68-6e0f-4430-989f-cb569677d74c": "How can we distinguish between fact and opinion in the content generated by AI systems?", "6fba0797-2aaa-4686-9325-999b5396f47b": "What are the risks associated with the anthropomorphization of General AI (GAI) systems?", "449ab90b-3762-4d3e-99ea-899bd340c42b": "What are confabulations in the context of text-based outputs?", "1c57be24-8e1d-4a3a-a29e-1d153c019510": "How do legal confabulations manifest in state-of-the-art language models?", "6b30e12e-cecf-4cd7-936e-84468c950a36": "What is the purpose of the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government?", "5547bf9b-ceae-4386-a486-7708637ab6a1": "What role do Navigators play according to HealthCaregov?", "a520c4cc-f2f6-4dd8-bd3a-a1a750440209": "What are the key principles outlined in the ISO/IEC Guide 71:2014 for addressing accessibility in standards?", "d72c0d17-abee-470b-8725-abf4aad59b3f": "How do the Web Content Accessibility Guidelines (WCAG) 20 impact web development practices?", "8e31e286-3ac3-488f-a211-4575fd663a17": "What are the key expectations for automated systems to ensure data privacy and protection from unchecked surveillance?", "76f71eb0-f3b8-425d-8772-65a5d214634f": "How can heightened oversight of surveillance systems prevent algorithmic discrimination based on community membership?", "eb21dff3-4dd0-47af-a449-b9b525386911": "What are the key considerations for ensuring equitable outcomes in fallback and escalation systems for automated systems?", "645a1801-9128-4977-947d-5437b8933966": "How can organizations ensure that human consideration and fallback mechanisms are conducted in a timely manner for automated systems?", "50321a04-5130-43ab-9305-cc1d548da8e0": "What are the extra protections for data related to sensitive domains like health and personal finance?", "32f6e506-6e82-41f7-b80c-f0702a537ca2": "How do technological developments impact the sensitivity of data domains and the need for enhanced data protections?", "7a80ac97-319d-452b-a900-e739da72ab44": "What are some benchmarks used to quantify systemic bias in GAI system outputs?", "bedb600e-b951-4e89-9442-24b971ff1b21": "How can fairness assessments help measure systemic bias in GAI systems?", "5fa67d29-3be8-4c81-a7c9-1a4d5dfa0ba7": "What are the potential biases in hiring tools that learn from a company's predominantly male employee base?", "868637a7-88fa-4891-bcfd-da1d37772744": "How do predictive models that use race as a factor affect Black students' academic guidance and major selection?", "6410524d-24f8-4aaf-8b70-5dcfc8272cd0": "What measures are being taken to prevent the misuse of Apple AirTags for stalking and harassment?", "d0990582-29f1-41c2-90e1-89c4efc58153": "How does crime prediction software perpetuate biases despite promises of being free from them?", "cbc06d96-6605-45f4-8067-0342ab04aac4": "What are the key elements to consider when incorporating GAI systems into an AI system inventory?", "b16b4f6b-0ec2-49bc-9453-9bbf1a8feea5": "How should organizations handle inventory exemptions for GAI systems embedded into application software?", "b1da9e4e-62f7-4d08-ac87-2b196fa9114e": "What measures can be taken to ensure automated systems protect against algorithmic discrimination?", "aa279423-ea2b-4fa2-beb1-7a6e1400c36f": "How can independent evaluations of automated systems be conducted without compromising individual privacy?", "8a9ae766-2f74-4272-bd84-e95787e5e943": "What are the best practices for determining data origin and content lineage in AI systems?", "2567081e-89ba-4d98-a746-eaf8503e5c5d": "How can test and evaluation processes be instituted for data and content flows within an AI system?", "fa585f44-6fb6-443b-983e-6304d9c2f5e1": "What are the expectations for automated systems in high-risk settings like criminal justice?", "f51b4b1e-689f-4a47-82a2-d9a9a0d30ab7": "How should the level of risk influence the design of explanatory mechanisms in automated systems?", "ecb05eb6-335e-4451-bf2e-4c8ad8e800bf": "What are the current methods for reporting AI incidents?", "51d512f0-8849-48de-a188-5aab8ddee724": "How do publicly available databases decide which AI incidents to track?", "80d2e492-0668-4e82-b83e-d1cef2355444": "What is the NIST AI 600-1 framework about?", "b7e8353e-ffb6-41a5-a321-d6b5521a03d5": "How does the NIST Trustworthy and Responsible AI framework address generative artificial intelligence risks?", "acbfd37b-65e1-440b-b4b1-9b3ee9a15fac": "What are the best practices for establishing acceptable use policies for GAI in human-AI teaming settings?", "c86085f1-bc71-4d66-8869-d5335b328ec7": "How can organizations effectively implement synthetic content detection and labeling tools?"}, "relevant_contexts": {"a188ff31-68db-46bc-b419-ac0fdecc6b1b": ["1eebe549-0cfa-4adf-84b0-ed9a06656695"], "c9f79a82-8775-4461-ac2f-b7b4e8734e96": ["1eebe549-0cfa-4adf-84b0-ed9a06656695"], "80b370f2-6d21-4ff6-bc9b-763317af5321": ["bd7c4ee6-636c-4e73-8669-68ae8df8a0e8"], "d4e20011-f4f9-4de6-8453-8144d77ba3a7": ["bd7c4ee6-636c-4e73-8669-68ae8df8a0e8"], "12534c5f-eb68-41cb-a879-b908e95a65d0": ["96206509-2450-4808-b3db-0ad36b187bf3"], "11951856-b35d-4123-ab15-80c38cd82195": ["96206509-2450-4808-b3db-0ad36b187bf3"], "869c7e52-2b77-4144-af5f-d5beebc138f9": ["5b799f01-f51b-4867-8554-833805f3ab80"], "a310a614-4bd4-4302-ab42-37b6faa7d838": ["5b799f01-f51b-4867-8554-833805f3ab80"], "468bff4b-cdbb-489a-9cbd-95d9062f93dc": ["43309aea-4c65-4a8b-9dbb-ad2c5402ed13"], "3e5b4545-4ceb-424f-b519-16c3870dd543": ["43309aea-4c65-4a8b-9dbb-ad2c5402ed13"], "0f0eea1d-f05e-458e-b31a-e955629c7e44": ["4c75b2c9-d74b-46ad-b25f-e5b2bbba9a2f"], "6cc0bcc5-9c82-49a6-b118-3e333e70ee9e": ["4c75b2c9-d74b-46ad-b25f-e5b2bbba9a2f"], "4441faa1-8f27-4fc7-bea0-847caa1c1505": ["ca9ae4fc-a936-4dda-acea-192bc0206464"], "0db8fdee-99cf-47c6-9d8d-a85f3b294826": ["ca9ae4fc-a936-4dda-acea-192bc0206464"], "57db460e-0123-4edf-b7df-87a967a60c26": ["4b00025b-f3dc-41ec-b5e2-b4f77272ad81"], "48589831-4f3c-4bf6-9cb4-bc4277c489dd": ["4b00025b-f3dc-41ec-b5e2-b4f77272ad81"], "1df11168-7aa5-4b43-91df-c14c32f01440": ["054c9a30-d999-4ec7-a07e-200e0ac42d1f"], "2127b35f-68cd-4e5f-a669-a6a4bb532fa8": ["054c9a30-d999-4ec7-a07e-200e0ac42d1f"], "afefb290-48ec-450c-b530-5fe1b6c5340b": ["1b4221a5-1a5d-4193-b4c3-d0927768a090"], "eecbf085-2f16-45c4-ba65-35813ca84568": ["1b4221a5-1a5d-4193-b4c3-d0927768a090"], "43b6add5-244e-4c11-be3b-0944fecfa6b9": ["e2a458cd-3f14-4aad-ad1d-0efcae5d686c"], "37bbd6b6-d24a-4b73-a4f4-f532d8c1793a": ["e2a458cd-3f14-4aad-ad1d-0efcae5d686c"], "318fe73a-0591-41e8-b65e-925c71b2caab": ["380e7d12-ea58-4f2f-bc0c-4e04c176047d"], "56664bc2-0933-4e58-8d03-5c06b9d06c04": ["380e7d12-ea58-4f2f-bc0c-4e04c176047d"], "7f8b418c-6e85-4ab0-83db-b7ed7dc49a45": ["b73c4e8f-15b1-48df-b5d3-0dc244b5e44d"], "e81617a3-9609-4012-ba46-caa374c306de": ["b73c4e8f-15b1-48df-b5d3-0dc244b5e44d"], "054e5797-d024-41bd-8de9-983d038a8797": ["fac21c98-5e09-4073-8499-737a13a0eb2d"], "fdb81ad2-acf2-4aa4-b551-fe766d22f273": ["fac21c98-5e09-4073-8499-737a13a0eb2d"], "09a4ef32-a01e-4ca9-9bf6-4704e328ccef": ["0026669e-4953-4d6a-b1d9-ecfa12faec64"], "e314f460-f6e2-4d11-b612-d51529a9dee6": ["0026669e-4953-4d6a-b1d9-ecfa12faec64"], "741f5989-422f-4bc5-9f72-0f3b22bb4f25": ["1171bb5d-18a9-429e-8122-da09f3a0d9f2"], "19592c9a-0621-4629-bdfc-8a08f0d396b4": ["1171bb5d-18a9-429e-8122-da09f3a0d9f2"], "f95100da-f55f-4402-909d-fdde5cf17d25": ["d96f7e82-cc68-47f6-86d2-85aa141a8c9e"], "bb3e7970-5b1e-4e98-87ad-b30d33ff6a89": ["d96f7e82-cc68-47f6-86d2-85aa141a8c9e"], "796ffa10-1532-4fa1-b832-d8ee058d410d": ["d44e9dcd-c607-44be-8995-10b21aae83a5"], "2c38117e-4b2d-4553-b319-f4ba3997996e": ["d44e9dcd-c607-44be-8995-10b21aae83a5"], "3b9c9379-cc75-4b9d-a68a-dc6b0a48fd9c": ["a9851a96-2f0d-44d3-bc00-c23aaa41be72"], "1a02235f-7bf0-4e7e-8149-ab610eacb769": ["a9851a96-2f0d-44d3-bc00-c23aaa41be72"], "08cbf993-d60b-4982-bf84-140c29d30450": ["394ba34f-5572-41aa-9636-d1f9f550d321"], "e253b5ac-feb7-4116-9e68-d2c817da36a5": ["394ba34f-5572-41aa-9636-d1f9f550d321"], "0260750e-4f7d-4c1b-b0b7-ae4c36cc8fc3": ["1c921767-4d8e-42c2-b1b7-f1eef6154d6f"], "39863570-2d41-4d21-bde1-1afc78c157b0": ["1c921767-4d8e-42c2-b1b7-f1eef6154d6f"], "8b4fd9d7-e1d4-472e-bd34-35fa98299c07": ["e4a13b31-217a-46da-a63d-97fb166719a8"], "1f34befe-4432-419f-8465-066a0d82ff77": ["e4a13b31-217a-46da-a63d-97fb166719a8"], "3e35db8c-c1b3-4b9f-b6c0-a3fd6e52d2b0": ["64bede83-602b-4ecc-9aa8-b7e66674fcbf"], "0c3b35ca-f421-41e4-b016-8f367561acbe": ["64bede83-602b-4ecc-9aa8-b7e66674fcbf"], "d73655e4-93f0-41c5-b69e-814ff8189db8": ["9d624f3e-302d-4fcf-9a0e-5e84ce69a0e6"], "bb8b9729-d1b6-407d-9f0c-aa1bd62a8d78": ["9d624f3e-302d-4fcf-9a0e-5e84ce69a0e6"], "3f1dec42-4087-4e06-9e7e-491c96cdee67": ["24ba513e-4acb-465b-be49-00cb67405123"], "9a94cfb2-25b9-4aa8-94c5-987c53fa42bf": ["24ba513e-4acb-465b-be49-00cb67405123"], "f445449e-b75f-44ee-a819-018ad630bd35": ["fb71dcec-b23f-4f60-a695-56ecd3f315ac"], "58fd202f-0791-411a-9124-09381dbbad11": ["fb71dcec-b23f-4f60-a695-56ecd3f315ac"], "26ee0f55-a947-440f-b4bc-4b7def4e3545": ["ee208f32-1e0d-4e1e-a351-3417bbd87afb"], "dcb01564-a34f-42a7-ac6c-13764525a7d2": ["ee208f32-1e0d-4e1e-a351-3417bbd87afb"], "8e29d29a-fc98-4a6f-b42b-580fc084dd71": ["193dbafa-5c73-4b7a-9b65-0df439acb9d8"], "4bbc6d4b-6b67-4831-8bca-853eb46aec3a": ["193dbafa-5c73-4b7a-9b65-0df439acb9d8"], "919fdd1d-2abb-472e-ac8d-bde9df2bb391": ["b115198f-f69a-4ce2-aebb-b3842c8f5271"], "a795e873-419b-454b-8598-fb0c49a7e5cc": ["b115198f-f69a-4ce2-aebb-b3842c8f5271"], "242b750e-1236-41f7-a1cc-eedef8f0427d": ["ad125822-a8be-416c-904e-df009ec77b21"], "c506e557-776f-42ed-99f9-c752ac2bb94b": ["ad125822-a8be-416c-904e-df009ec77b21"], "426616e2-6297-47c3-89c7-71ec1186cdba": ["e44738ee-74b6-4246-bc14-d817afb94e83"], "db43af55-434d-441f-8dc7-acc8ff3f8432": ["e44738ee-74b6-4246-bc14-d817afb94e83"], "f2913868-28a6-4558-904a-0486fbfc1f6e": ["68ce524c-132f-488c-adcf-6d6b0fd3ee28"], "5a4faa70-0364-4fd0-9c98-b26fb63f7786": ["68ce524c-132f-488c-adcf-6d6b0fd3ee28"], "3ad57490-e4f7-4fd2-bff4-93211043ec13": ["ed722cdb-468f-4721-a373-d1ca5a35c1f9"], "3a05d7ba-2e46-406b-aeb9-51b33efff15f": ["ed722cdb-468f-4721-a373-d1ca5a35c1f9"], "6ae09ea8-3090-401b-9f1e-4ce5270152cd": ["4097f22e-c5bf-4c18-8078-c3a2899b5bfb"], "207207ff-faab-4342-b76f-ef0c6fac88c9": ["4097f22e-c5bf-4c18-8078-c3a2899b5bfb"], "d15a10aa-36cb-4f3a-9f9e-2c0416ce1084": ["72d14b3e-b07e-43bd-9020-1a2c23f4ef52"], "c9f4fb11-9365-4354-aa94-7cc93efcafb5": ["72d14b3e-b07e-43bd-9020-1a2c23f4ef52"], "4aebac20-11d4-42c8-be6a-f7ac4e43cbbc": ["db18094e-cd82-4e21-8d23-3a29d290999b"], "71537f88-7e77-4720-9cc0-bca516b4721f": ["db18094e-cd82-4e21-8d23-3a29d290999b"], "e004e796-65d5-4109-89bd-472cae5b6c75": ["094c20fa-14b1-497b-b40e-5b99c32cf2fc"], "290cd0b2-456b-41bc-bf0e-3ea3e32f480d": ["094c20fa-14b1-497b-b40e-5b99c32cf2fc"], "1c416614-5e28-45f4-9e8e-937971dcff9a": ["f33bc6b2-858a-46bd-ba56-b6410ce7b11b"], "d83ab93d-9be0-488d-94fd-8e58074a3388": ["f33bc6b2-858a-46bd-ba56-b6410ce7b11b"], "bfc45e93-d073-4348-8fb1-03dfaf4e73f3": ["ea01c2f2-4936-4233-8845-855c033c5a09"], "4819bdb4-1724-4318-855c-9c4f680c0655": ["ea01c2f2-4936-4233-8845-855c033c5a09"], "a8a96840-d387-42d9-9b56-f05b73027f5c": ["641dd569-3b6d-49b4-ab74-5b743949ed5d"], "7fdb6c15-c3f8-4327-b2fe-0169c08ce375": ["641dd569-3b6d-49b4-ab74-5b743949ed5d"], "3509c40f-7af0-49a5-bd16-c7da584b3980": ["ea99d79c-dacc-4993-a145-2146a1469e05"], "a86eba64-72a8-4afa-a7f5-8c50c3b0c660": ["ea99d79c-dacc-4993-a145-2146a1469e05"], "9eee9d68-6e0f-4430-989f-cb569677d74c": ["e8a4ecfe-f6e5-4984-8f0c-694996adfb03"], "6fba0797-2aaa-4686-9325-999b5396f47b": ["e8a4ecfe-f6e5-4984-8f0c-694996adfb03"], "449ab90b-3762-4d3e-99ea-899bd340c42b": ["a7b25bc5-d04c-4ce5-b11d-18080ed7322b"], "1c57be24-8e1d-4a3a-a29e-1d153c019510": ["a7b25bc5-d04c-4ce5-b11d-18080ed7322b"], "6b30e12e-cecf-4cd7-936e-84468c950a36": ["0422346b-f47b-48ad-890e-93045e292363"], "5547bf9b-ceae-4386-a486-7708637ab6a1": ["0422346b-f47b-48ad-890e-93045e292363"], "a520c4cc-f2f6-4dd8-bd3a-a1a750440209": ["d444272b-84db-47b2-8e39-d070bef54d11"], "d72c0d17-abee-470b-8725-abf4aad59b3f": ["d444272b-84db-47b2-8e39-d070bef54d11"], "8e31e286-3ac3-488f-a211-4575fd663a17": ["84e5065a-6f26-49c3-aeb8-31a8102a856b"], "76f71eb0-f3b8-425d-8772-65a5d214634f": ["84e5065a-6f26-49c3-aeb8-31a8102a856b"], "eb21dff3-4dd0-47af-a449-b9b525386911": ["3976a13c-4484-47bc-8b1d-0fcb75a19b95"], "645a1801-9128-4977-947d-5437b8933966": ["3976a13c-4484-47bc-8b1d-0fcb75a19b95"], "50321a04-5130-43ab-9305-cc1d548da8e0": ["88018024-6cf6-4719-ad61-61f79483bb74"], "32f6e506-6e82-41f7-b80c-f0702a537ca2": ["88018024-6cf6-4719-ad61-61f79483bb74"], "7a80ac97-319d-452b-a900-e739da72ab44": ["641be3b7-f879-4cc0-bc16-d9cb27069618"], "bedb600e-b951-4e89-9442-24b971ff1b21": ["641be3b7-f879-4cc0-bc16-d9cb27069618"], "5fa67d29-3be8-4c81-a7c9-1a4d5dfa0ba7": ["f12b5467-1c94-4938-98a8-5e0e4e6fff77"], "868637a7-88fa-4891-bcfd-da1d37772744": ["f12b5467-1c94-4938-98a8-5e0e4e6fff77"], "6410524d-24f8-4aaf-8b70-5dcfc8272cd0": ["380caf5a-f592-4a9d-8e55-905836b69ded"], "d0990582-29f1-41c2-90e1-89c4efc58153": ["380caf5a-f592-4a9d-8e55-905836b69ded"], "cbc06d96-6605-45f4-8067-0342ab04aac4": ["5b9ba636-3418-4270-a189-27f4e5b95ae0"], "b16b4f6b-0ec2-49bc-9453-9bbf1a8feea5": ["5b9ba636-3418-4270-a189-27f4e5b95ae0"], "b1da9e4e-62f7-4d08-ac87-2b196fa9114e": ["c3f7bcbe-0afe-4e8b-a6c2-8266ee6bec0a"], "aa279423-ea2b-4fa2-beb1-7a6e1400c36f": ["c3f7bcbe-0afe-4e8b-a6c2-8266ee6bec0a"], "8a9ae766-2f74-4272-bd84-e95787e5e943": ["f78abfc0-dc1b-4904-b10f-45b2d75bdffa"], "2567081e-89ba-4d98-a746-eaf8503e5c5d": ["f78abfc0-dc1b-4904-b10f-45b2d75bdffa"], "fa585f44-6fb6-443b-983e-6304d9c2f5e1": ["e88db2aa-0248-4c41-9ff5-f64b062d93ad"], "f51b4b1e-689f-4a47-82a2-d9a9a0d30ab7": ["e88db2aa-0248-4c41-9ff5-f64b062d93ad"], "ecb05eb6-335e-4451-bf2e-4c8ad8e800bf": ["481dbfa9-e17c-4a32-bfda-547eb5403563"], "51d512f0-8849-48de-a188-5aab8ddee724": ["481dbfa9-e17c-4a32-bfda-547eb5403563"], "80d2e492-0668-4e82-b83e-d1cef2355444": ["60edd255-562c-403c-b6b1-20d1d828e53f"], "b7e8353e-ffb6-41a5-a321-d6b5521a03d5": ["60edd255-562c-403c-b6b1-20d1d828e53f"], "acbfd37b-65e1-440b-b4b1-9b3ee9a15fac": ["810d4e10-aa6e-4399-aee2-0740c4dc03c4"], "c86085f1-bc71-4d66-8869-d5335b328ec7": ["810d4e10-aa6e-4399-aee2-0740c4dc03c4"]}, "corpus": {"1eebe549-0cfa-4adf-84b0-ed9a06656695": "Another cybersecurity risk to GAI is data poisoning, in which an adversary compromises a training \ndataset used by a model to manipulate its outputs or operation. Malicious tampering with data or parts \nof the model could exacerbate risks associated with GAI system outputs. \nTrustworthy AI Characteristics: Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable \n2.10. \nIntellectual Property \nIntellectual property risks from GAI systems may arise where the use of copyrighted works is not a fair \nuse under the fair use doctrine. If a GAI system\u2019s training data included copyrighted material, GAI \noutputs displaying instances of training data memorization (see Data Privacy above) could infringe on \ncopyright. \nHow GAI relates to copyright, including the status of generated content that is similar to but does not \nstrictly copy work protected by copyright, is currently being debated in legal fora. Similar discussions are", "bd7c4ee6-636c-4e73-8669-68ae8df8a0e8": "or lead to algorithmic discrimination. \nOversight. Human-based systems have the potential for bias, including automation bias, as well as other \nconcerns that may limit their effectiveness. The results of assessments of the efficacy and potential bias of \nsuch human-based systems should be overseen by governance structures that have the potential to update the \noperation of the human-based system in order to mitigate these effects. \n50", "96206509-2450-4808-b3db-0ad36b187bf3": "Intellectual Property; Data Privacy; \nObscene, Degrading, and/or \nAbusive Content \nMP-4.1-005 \nEstablish policies for collection, retention, and minimum quality of data, in \nconsideration of the following risks: Disclosure of inappropriate CBRN information; \nUse of Illegal or dangerous content; O\ufb00ensive cyber capabilities; Training data \nimbalances that could give rise to harmful biases; Leak of personally identi\ufb01able \ninformation, including facial likenesses of individuals. \nCBRN Information or Capabilities; \nIntellectual Property; Information \nSecurity; Harmful Bias and \nHomogenization; Dangerous, \nViolent, or Hateful Content; Data \nPrivacy \nMP-4.1-006 Implement policies and practices de\ufb01ning how third-party intellectual property and \ntraining data will be used, stored, and protected. \nIntellectual Property; Value Chain \nand Component Integration \nMP-4.1-007 Re-evaluate models that were \ufb01ne-tuned or enhanced on top of third-party \nmodels. \nValue Chain and Component \nIntegration \nMP-4.1-008", "5b799f01-f51b-4867-8554-833805f3ab80": "SAFE AND EFFECTIVE \nSYSTEMS \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nIn order to ensure that an automated system is safe and effective, it should include safeguards to protect the \npublic from harm in a proactive and ongoing manner; avoid use of data inappropriate for or irrelevant to the task \nat hand, including reuse that could cause compounded harm; and demonstrate the safety and effectiveness of \nthe system. These expectations are explained below. \nProtect the public from harm in a proactive and ongoing manner \nConsultation. The public should be consulted in the design, implementation, deployment, acquisition, and \nmaintenance phases of automated system development, with emphasis on early-stage consultation before a", "43309aea-4c65-4a8b-9dbb-ad2c5402ed13": "\u2022\nSurya Mattu, Senior Data Engineer and Investigative Data Journalist, The Markup\n\u2022\nMariah Montgomery, National Campaign Director, Partnership for Working Families\n55", "4c75b2c9-d74b-46ad-b25f-e5b2bbba9a2f": "While indirect feedback methods such as automated error collection systems are useful, they often lack \nthe context and depth that direct input from end users can provide. Organizations can leverage feedback \napproaches described in the Pre-Deployment Testing section to capture input from external sources such \nas through AI red-teaming.  \nIntegrating pre- and post-deployment external feedback into the monitoring process for GAI models and \ncorresponding applications can help enhance awareness of performance changes and mitigate potential \nrisks and harms from outputs. There are many ways to capture and make use of user feedback \u2013 before \nand after GAI systems and digital content transparency approaches are deployed \u2013 to gain insights about \nauthentication e\ufb03cacy and vulnerabilities, impacts of adversarial threats on techniques, and unintended \nconsequences resulting from the utilization of content provenance approaches on users and", "ca9ae4fc-a936-4dda-acea-192bc0206464": "technology may or may not be part of an effective set of mechanisms to achieve safety. Various panelists raised \nconcerns about the validity of these systems, the tendency of adverse or irrelevant data to lead to a replication of \nunjust outcomes, and the confirmation bias and tendency of people to defer to potentially inaccurate automated \nsystems. Throughout, many of the panelists individually emphasized that the impact of these systems on \nindividuals and communities is potentially severe: the systems lack individualization and work against the \nbelief that people can change for the better, system use can lead to the loss of jobs and custody of children, and \nsurveillance can lead to chilling effects for communities and sends negative signals to community members \nabout how they're viewed. \nIn discussion of technical and governance interventions that that are needed to protect against the harms of", "4b00025b-f3dc-41ec-b5e2-b4f77272ad81": "32 \nMEASURE 2.6: The AI system is evaluated regularly for safety risks \u2013 as identi\ufb01ed in the MAP function. The AI system to be \ndeployed is demonstrated to be safe, its residual negative risk does not exceed the risk tolerance, and it can fail safely, particularly if \nmade to operate beyond its knowledge limits. Safety metrics re\ufb02ect system reliability and robustness, real-time monitoring, and \nresponse times for AI system failures. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.6-001 \nAssess adverse impacts, including health and wellbeing impacts for value chain \nor other AI Actors that are exposed to sexually explicit, o\ufb00ensive, or violent \ninformation during GAI training and maintenance. \nHuman-AI Con\ufb01guration; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and \nComponent Integration; \nDangerous, Violent, or Hateful \nContent \nMS-2.6-002 \nAssess existence or levels of harmful bias, intellectual property infringement,", "054c9a30-d999-4ec7-a07e-200e0ac42d1f": "into other automated systems that directly impact people\u2019s lives. Federal law has not grown to address the expanding \nscale of private data collection, or of the ability of governments at all levels to access that data and leverage the means \nof private collection.  \nMeanwhile, members of the American public are often unable to access their personal data or make critical decisions \nabout its collection and use. Data brokers frequently collect consumer data from numerous sources without \nconsumers\u2019 permission or knowledge.60 Moreover, there is a risk that inaccurate and faulty data can be used to \nmake decisions about their lives, such as whether they will qualify for a loan or get a job. Use of surveillance \ntechnologies has increased in schools and workplaces, and, when coupled with consequential management and \nevaluation decisions, it is leading to mental health harms such as lowered self-confidence, anxiety, depression, and", "1b4221a5-1a5d-4193-b4c3-d0927768a090": "110. Rachel Orey and Owen Bacskai. The Low Down on Ballot Curing. Nov. 04, 2020. https://\nbipartisanpolicy.org/blog/the-low-down-on-ballot-curing/; Zahavah Levine and Thea Raymond-\nSeidel. Mail Voting Litigation in 2020, Part IV: Verifying Mail Ballots. Oct. 29, 2020.\nhttps://www.lawfareblog.com/mail-voting-litigation-2020-part-iv-verifying-mail-ballots\n111. National Conference of State Legislatures. Table 15: States With Signature Cure Processes. Jan. 18,\n2022.\nhttps://www.ncsl.org/research/elections-and-campaigns/vopp-table-15-states-that-permit-voters-to\u00ad\ncorrect-signature-discrepancies.aspx\n112. White House Office of Science and Technology Policy. Join the Effort to Create A Bill of Rights for\nan Automated Society. Nov. 10, 2021.\nhttps://www.whitehouse.gov/ostp/news-updates/2021/11/10/join-the-effort-to-create-a-bill-of\u00ad\nrights-for-an-automated-society/\n113. White House Office of Science and Technology Policy. Notice of Request for Information (RFI) on", "e2a458cd-3f14-4aad-ad1d-0efcae5d686c": "Research Institute Report. June 29, 2021. https://datasociety.net/library/assembling-accountability\u00ad\nalgorithmic-impact-assessment-for-the-public-interest/; Nicol Turner Lee, Paul Resnick, and Genie\nBarton. Algorithmic bias detection and mitigation: Best practices and policies to reduce consumer harms.\nBrookings Report. May 22, 2019.\nhttps://www.brookings.edu/research/algorithmic-bias-detection-and-mitigation-best-practices-and\u00ad\npolicies-to-reduce-consumer-harms/; Andrew D. Selbst. An Institutional View Of Algorithmic Impact\nAssessments. Harvard Journal of Law & Technology. June 15, 2021. https://ssrn.com/abstract=3867634;\nDillon Reisman, Jason Schultz, Kate Crawford, and Meredith Whittaker. Algorithmic Impact\nAssessments: A Practical Framework for Public Agency Accountability. AI Now Institute Report. April\n2018. https://ainowinstitute.org/aiareport2018.pdf\n51. Department of Justice. Justice Department Announces New Initiative to Combat Redlining. Oct. 22,", "380e7d12-ea58-4f2f-bc0c-4e04c176047d": "HOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \nThe federal government is working to combat discrimination in mortgage lending. The Depart\u00ad\nment of Justice has launched a nationwide initiative to combat redlining, which includes reviewing how \nlenders who may be avoiding serving communities of color are conducting targeted marketing and advertising.51 \nThis initiative will draw upon strong partnerships across federal agencies, including the Consumer Financial \nProtection Bureau and prudential regulators. The Action Plan to Advance Property Appraisal and Valuation \nEquity includes a commitment from the agencies that oversee mortgage lending to include a \nnondiscrimination standard in the proposed rules for Automated Valuation Models.52", "b73c4e8f-15b1-48df-b5d3-0dc244b5e44d": "Intellectual Property \nGV-6.1-009 \nUpdate and integrate due diligence processes for GAI acquisition and \nprocurement vendor assessments to include intellectual property, data privacy, \nsecurity, and other risks. For example, update processes to: Address solutions that \nmay rely on embedded GAI technologies; Address ongoing monitoring, \nassessments, and alerting, dynamic risk assessments, and real-time reporting \ntools for monitoring third-party GAI risks; Consider policy adjustments across GAI \nmodeling libraries, tools and APIs, \ufb01ne-tuned models, and embedded tools; \nAssess GAI vendors, open-source or proprietary GAI tools, or GAI service \nproviders against incident or vulnerability databases. \nData Privacy; Human-AI \nCon\ufb01guration; Information \nSecurity; Intellectual Property; \nValue Chain and Component \nIntegration; Harmful Bias and \nHomogenization \nGV-6.1-010 \nUpdate GAI acceptable use policies to address proprietary and open-source GAI", "fac21c98-5e09-4073-8499-737a13a0eb2d": "disparity testing results and mitigation information, should be performed and made public whenever \npossible to confirm these protections. \n5", "0026669e-4953-4d6a-b1d9-ecfa12faec64": "\u2022\nA device originally developed to help people track and find lost items has been used as a tool by stalkers to track\nvictims\u2019 locations in violation of their privacy and safety. The device manufacturer took steps after release to\nprotect people from unwanted tracking by alerting people on their phones when a device is found to be moving\nwith them over time and also by having the device make an occasional noise, but not all phones are able\nto receive the notification and the devices remain a safety concern due to their misuse.8 \n\u2022\nAn algorithm used to deploy police was found to repeatedly send police to neighborhoods they regularly visit,\neven if those neighborhoods were not the ones with the highest crime rates. These incorrect crime predictions\nwere the result of a feedback loop generated from the reuse of data from previous arrests and algorithm\npredictions.9\n16", "1171bb5d-18a9-429e-8122-da09f3a0d9f2": "the creation and spread of NCII disproportionately impacts women and sexual minorities, and can have \nsubsequent negative consequences including decline in overall mental health, substance abuse, and \neven suicidal thoughts.  \nData used for training GAI models may unintentionally include CSAM and NCII. A recent report noted \nthat several commonly used GAI training datasets were found to contain hundreds of known images of", "d96f7e82-cc68-47f6-86d2-85aa141a8c9e": "APPENDIX\nPanel 4: Artificial Intelligence and Democratic Values. This event examined challenges and opportunities in \nthe design of technology that can help support a democratic vision for AI. It included discussion of the \ntechnical aspects \nof \ndesigning \nnon-discriminatory \ntechnology, \nexplainable \nAI, \nhuman-computer \ninteraction with an emphasis on community participation, and privacy-aware design. \nWelcome:\n\u2022\nSorelle Friedler, Assistant Director for Data and Democracy, White House Office of Science and\nTechnology Policy\n\u2022\nJ. Bob Alotta, Vice President for Global Programs, Mozilla Foundation\n\u2022\nNavrina Singh, Board Member, Mozilla Foundation\nModerator: Kathy Pham Evans, Deputy Chief Technology Officer for Product and Engineering, U.S \nFederal Trade Commission. \nPanelists: \n\u2022\nLiz O\u2019Sullivan, CEO, Parity AI\n\u2022\nTimnit Gebru, Independent Scholar\n\u2022\nJennifer Wortman Vaughan, Senior Principal Researcher, Microsoft Research, New York City\n\u2022", "d44e9dcd-c607-44be-8995-10b21aae83a5": "58 \nSatariano, A. et al. (2023) The People Onscreen Are Fake. The Disinformation Is Real. New York Times. \nhttps://www.nytimes.com/2023/02/07/technology/arti\ufb01cial-intelligence-training-deepfake.html \nSchaul, K. et al. (2024) Inside the secret list of websites that make AI like ChatGPT sound smart. \nWashington Post. https://www.washingtonpost.com/technology/interactive/2023/ai-chatbot-learning/ \nScheurer, J. et al. (2023) Technical report: Large language models can strategically deceive their users \nwhen put under pressure. arXiv. https://arxiv.org/abs/2311.07590 \nShelby, R. et al. (2023) Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm \nReduction. arXiv. https://arxiv.org/pdf/2210.05791 \nShevlane, T. et al. (2023) Model evaluation for extreme risks. arXiv. https://arxiv.org/pdf/2305.15324 \nShumailov, I. et al. (2023) The curse of recursion: training on generated data makes models forget. arXiv. \nhttps://arxiv.org/pdf/2305.17493v2", "a9851a96-2f0d-44d3-bc00-c23aaa41be72": "In addition to the suggested actions below, AI risk management activities and actions set forth in the AI \nRMF 1.0 and Playbook are already applicable for managing GAI risks. Organizations are encouraged to \napply the activities suggested in the AI RMF and its Playbook when managing the risk of GAI systems.  \nImplementation of the suggested actions will vary depending on the type of risk, characteristics of GAI \nsystems, stage of the GAI lifecycle, and relevant AI actors involved.  \nSuggested actions to manage GAI risks can be found in the tables below: \n\u2022 \nThe suggested actions are organized by relevant AI RMF subcategories to streamline these \nactivities alongside implementation of the AI RMF.  \n\u2022 \nNot every subcategory of the AI RMF is included in this document.13 Suggested actions are \nlisted for only some subcategories.  \n \n \n13 As this document was focused on the GAI PWG e\ufb00orts and primary considerations (see Appendix A), AI RMF \nsubcategories not addressed here may be added later.", "394ba34f-5572-41aa-9636-d1f9f550d321": "Provide the public with mechanisms for appropriate and meaningful consent, access, and \ncontrol over their data \nUse-specific consent. Consent practices should not allow for abusive surveillance practices. Where data \ncollectors or automated systems seek consent, they should seek it for specific, narrow use contexts, for specif\u00ad\nic time durations, and for use by specific entities. Consent should not extend if any of these conditions change; \nconsent should be re-acquired before using data if the use case changes, a time limit elapses, or data is trans\u00ad\nferred to another entity (including being shared or sold). Consent requested should be limited in scope and \nshould not request consent beyond what is required. Refusal to provide consent should be allowed, without \nadverse effects, to the greatest extent possible based on the needs of the use case. \nBrief and direct consent requests. When seeking consent from users short, plain language consent", "1c921767-4d8e-42c2-b1b7-f1eef6154d6f": "TABLE OF CONTENTS\nFROM PRINCIPLES TO PRACTICE: A TECHNICAL COMPANION TO THE BLUEPRINT \nFOR AN AI BILL OF RIGHTS \n \nUSING THIS TECHNICAL COMPANION\n \nSAFE AND EFFECTIVE SYSTEMS\n \nALGORITHMIC DISCRIMINATION PROTECTIONS\n \nDATA PRIVACY\n \nNOTICE AND EXPLANATION\n \nHUMAN ALTERNATIVES, CONSIDERATION, AND FALLBACK\nAPPENDIX\n \nEXAMPLES OF AUTOMATED SYSTEMS\n \nLISTENING TO THE AMERICAN PEOPLE\nENDNOTES \n12\n14\n15\n23\n30\n40\n46\n53\n53\n55\n63\n13", "e4a13b31-217a-46da-a63d-97fb166719a8": "Human-AI Con\ufb01guration \nMS-2.5-003 Review and verify sources and citations in GAI system outputs during pre-\ndeployment risk measurement and ongoing monitoring activities. \nConfabulation \nMS-2.5-004 Track and document instances of anthropomorphization (e.g., human images, \nmentions of human feelings, cyborg imagery or motifs) in GAI system interfaces. Human-AI Con\ufb01guration \nMS-2.5-005 Verify GAI system training data and TEVV data provenance, and that \ufb01ne-tuning \nor retrieval-augmented generation data is grounded. \nInformation Integrity \nMS-2.5-006 \nRegularly review security and safety guardrails, especially if the GAI system is \nbeing operated in novel circumstances. This includes reviewing reasons why the \nGAI system was initially assessed as being safe to deploy.  \nInformation Security; Dangerous, \nViolent, or Hateful Content \nAI Actor Tasks: Domain Experts, TEVV", "64bede83-602b-4ecc-9aa8-b7e66674fcbf": "public; this section focuses on red-teaming in pre-deployment contexts.  \nThe quality of AI red-teaming outputs is related to the background and expertise of the AI red team \nitself. Demographically and interdisciplinarily diverse AI red teams can be used to identify \ufb02aws in the \nvarying contexts where GAI will be used. For best results, AI red teams should demonstrate domain \nexpertise, and awareness of socio-cultural aspects within the deployment context. AI red-teaming results \nshould be given additional analysis before they are incorporated into organizational governance and \ndecision making, policy and procedural updates, and AI risk management e\ufb00orts. \nVarious types of AI red-teaming may be appropriate, depending on the use case: \n\u2022 \nGeneral Public: Performed by general users (not necessarily AI or technical experts) who are \nexpected to use the model or interact with its outputs, and who bring their own lived \nexperiences and perspectives to the task of AI red-teaming. These individuals may have been", "9d624f3e-302d-4fcf-9a0e-5e84ce69a0e6": "34. Todd Feathers. Major Universities Are Using Race as a \u201cHigh Impact Predictor\u201d of Student Success:\nStudents, professors, and education experts worry that that\u2019s pushing Black students in particular out of math\nand science. The Markup. Mar. 2, 2021. https://themarkup.org/machine-learning/2021/03/02/major\u00ad\nuniversities-are-using-race-as-a-high-impact-predictor-of-student-success\n65", "24ba513e-4acb-465b-be49-00cb67405123": "APPENDIX\nPanelists discussed the benefits of AI-enabled systems and their potential to build better and more \ninnovative infrastructure. They individually noted that while AI technologies may be new, the process of \ntechnological diffusion is not, and that it was critical to have thoughtful and responsible development and \nintegration of technology within communities. Some panelists suggested that the integration of technology \ncould benefit from examining how technological diffusion has worked in the realm of urban planning: \nlessons learned from successes and failures there include the importance of balancing ownership rights, use \nrights, and community health, safety and welfare, as well ensuring better representation of all voices, \nespecially those traditionally marginalized by technological advances. Some panelists also raised the issue of \npower structures \u2013 providing examples of how strong transparency requirements in smart city projects", "fb71dcec-b23f-4f60-a695-56ecd3f315ac": "SECTION TITLE\nHUMAN ALTERNATIVES, CONSIDERATION, AND FALLBACK\nYou should be able to opt out, where appropriate, and have access to a person who can quickly \nconsider and remedy problems you encounter. You should be able to opt out from automated systems in \nfavor of a human alternative, where appropriate. Appropriateness should be determined based on reasonable \nexpectations in a given context and with a focus on ensuring broad accessibility and protecting the public from \nespecially harmful impacts. In some cases, a human or other alternative may be required by law. You should have \naccess to timely human consideration and remedy by a fallback and escalation process if an automated system \nfails, it produces an error, or you would like to appeal or contest its impacts on you. Human consideration and \nfallback should be accessible, equitable, effective, maintained, accompanied by appropriate operator training, and", "ee208f32-1e0d-4e1e-a351-3417bbd87afb": "\u2022\nA risk assessment tool designed to predict the risk of recidivism for individuals in federal custody showed\nevidence of disparity in prediction. The tool overpredicts the risk of recidivism for some groups of color on the\ngeneral recidivism tools, and underpredicts the risk of recidivism for some groups of color on some of the\nviolent recidivism tools. The Department of Justice is working to reduce these disparities and has\npublicly released a report detailing its review of the tool.35 \n24", "193dbafa-5c73-4b7a-9b65-0df439acb9d8": "ENDNOTES\n1.The Executive Order On Advancing Racial Equity and Support for Underserved Communities Through the\nFederal\u00a0Government. https://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive\norder-advancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/\n2. The White House. Remarks by President Biden on the Supreme Court Decision to Overturn Roe v. Wade. Jun.\n24, 2022. https://www.whitehouse.gov/briefing-room/speeches-remarks/2022/06/24/remarks-by-president\u00ad\nbiden-on-the-supreme-court-decision-to-overturn-roe-v-wade/\n3. The White House. Join the Effort to Create A Bill of Rights for an Automated Society. Nov. 10, 2021. https://\nwww.whitehouse.gov/ostp/news-updates/2021/11/10/join-the-effort-to-create-a-bill-of-rights-for-an\u00ad\nautomated-society/\n4. U.S. Dept. of Health, Educ. & Welfare, Report of the Sec\u2019y\u2019s Advisory Comm. on Automated Pers. Data Sys.,", "b115198f-f69a-4ce2-aebb-b3842c8f5271": "Value Chain and Component \nIntegration; Harmful Bias and \nHomogenization \nMG-3.1-003 \nRe-assess model risks after \ufb01ne-tuning or retrieval-augmented generation \nimplementation and for any third-party GAI models deployed for applications \nand/or use cases that were not evaluated in initial testing. \nValue Chain and Component \nIntegration \nMG-3.1-004 \nTake reasonable measures to review training data for CBRN information, and \nintellectual property, and where appropriate, remove it. Implement reasonable \nmeasures to prevent, \ufb02ag, or take other action in response to outputs that \nreproduce particular training data (e.g., plagiarized, trademarked, patented, \nlicensed content or trade secret material). \nIntellectual Property; CBRN \nInformation or Capabilities", "ad125822-a8be-416c-904e-df009ec77b21": "communities. Furthermore, organizations can track and document the provenance of datasets to identify \ninstances in which AI-generated data is a potential root cause of performance issues with the GAI \nsystem. \nA.1.8. Incident Disclosure \nOverview \nAI incidents can be de\ufb01ned as an \u201cevent, circumstance, or series of events where the development, use, \nor malfunction of one or more AI systems directly or indirectly contributes to one of the following harms: \ninjury or harm to the health of a person or groups of people (including psychological harms and harms to \nmental health); disruption of the management and operation of critical infrastructure; violations of \nhuman rights or a breach of obligations under applicable law intended to protect fundamental, labor, \nand intellectual property rights; or harm to property, communities, or the environment.\u201d AI incidents can \noccur in the aggregate (i.e., for systemic discrimination) or acutely (i.e., for one individual). \nState of AI Incident Tracking and Disclosure", "e44738ee-74b6-4246-bc14-d817afb94e83": "American Civil Liberties Union \nAmerican Civil Liberties Union of \nMassachusetts \nAmerican Medical Association \nARTICLE19 \nAttorneys General of the District of \nColumbia, Illinois, Maryland, \nMichigan, Minnesota, New York, \nNorth Carolina, Oregon, Vermont, \nand Washington \nAvanade \nAware \nBarbara Evans \nBetter Identity Coalition \nBipartisan Policy Center \nBrandon L. Garrett and Cynthia \nRudin \nBrian Krupp \nBrooklyn Defender Services \nBSA | The Software Alliance \nCarnegie Mellon University \nCenter for Democracy & \nTechnology \nCenter for New Democratic \nProcesses \nCenter for Research and Education \non Accessible Technology and \nExperiences at University of \nWashington, Devva Kasnitz, L Jean \nCamp, Jonathan Lazar, Harry \nHochheiser \nCenter on Privacy & Technology at \nGeorgetown Law \nCisco Systems \nCity of Portland Smart City PDX \nProgram \nCLEAR \nClearview AI \nCognoa \nColor of Change \nCommon Sense Media \nComputing Community Consortium \nat Computing Research Association \nConnected Health Initiative", "68ce524c-132f-488c-adcf-6d6b0fd3ee28": "ing should take into account the performance of both technical system components (the algorithm as well as \nany hardware components, data inputs, etc.) and human operators. It should include mechanisms for testing \nthe actual accuracy of any predictions or recommendations generated by a system, not just a human operator\u2019s \ndetermination of their accuracy. Ongoing monitoring procedures should include manual, human-led monitor\u00ad\ning as a check in the event there are shortcomings in automated monitoring systems. These monitoring proce\u00ad\ndures should be in place for the lifespan of the deployed automated system. \nClear organizational oversight. Entities responsible for the development or use of automated systems \nshould lay out clear governance structures and procedures.  This includes clearly-stated governance proce\u00ad\ndures before deploying the system, as well as responsibility of specific individuals or entities to oversee ongoing", "ed722cdb-468f-4721-a373-d1ca5a35c1f9": "should not impose an unreasonable burden on the public. Automated systems with an intended use within sensi\u00ad\ntive domains, including, but not limited to, criminal justice, employment, education, and health, should additional\u00ad\nly be tailored to the purpose, provide meaningful access for oversight, include training for any people interacting \nwith the system, and incorporate human consideration for adverse or high-risk decisions. Reporting that includes \na description of these human governance processes and assessment of their timeliness, accessibility, outcomes, \nand effectiveness should be made public whenever possible. \nDefinitions for key terms in The Blueprint for an AI Bill of Rights can be found in Applying the Blueprint for an AI Bill of Rights. \nAccompanying analysis and tools for actualizing each principle can be found in the Technical Companion. \n7", "4097f22e-c5bf-4c18-8078-c3a2899b5bfb": "DATA PRIVACY \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \u00ad\u00ad\u00ad\u00ad\u00ad\u00ad\nIn addition to the privacy expectations above for general non-sensitive data, any system collecting, using, shar-\ning, or storing sensitive data should meet the expectations below. Depending on the technological use case and \nbased on an ethical assessment, consent for sensitive data may need to be acquired from a guardian and/or child. \nProvide enhanced protections for data related to sensitive domains \nNecessary functions only. Sensitive data should only be used for functions strictly necessary for that \ndomain or for functions that are required for administrative reasons (e.g., school attendance records), unless \nconsent is acquired, if appropriate, and the additional expectations in this section are met. Consent for non-", "72d14b3e-b07e-43bd-9020-1a2c23f4ef52": "researchers: Chloe Autio, Jesse Dunietz, Patrick Hall, Shomik Jain, Kamie Roberts, Reva Schwartz, Martin \nStanley, and Elham Tabassi. \nNIST Technical Series Policies \nCopyright, Use, and Licensing Statements \nNIST Technical Series Publication Identifier Syntax \nPublication History \nApproved by the NIST Editorial Review Board on 07-25-2024 \nContact Information \[email protected] \nNational Institute of Standards and Technology \nAttn: NIST AI Innovation Lab, Information Technology Laboratory \n100 Bureau Drive (Mail Stop 8900) Gaithersburg, MD 20899-8900 \nAdditional Information \nAdditional information about this publication and other NIST AI publications are available at \nhttps://airc.nist.gov/Home. \n \nDisclaimer: Certain commercial entities, equipment, or materials may be identi\ufb01ed in this document in \norder to adequately describe an experimental procedure or concept. Such identi\ufb01cation is not intended to \nimply recommendation or endorsement by the National Institute of Standards and Technology, nor is it", "db18094e-cd82-4e21-8d23-3a29d290999b": "should be maintained and supported as long as the relevant automated system continues to be in use. \nInstitute training, assessment, and oversight to combat automation bias and ensure any \nhuman-based components of a system are effective. \nTraining and assessment. Anyone administering, interacting with, or interpreting the outputs of an auto\u00ad\nmated system should receive training in that system, including how to properly interpret outputs of a system \nin light of its intended purpose and in how to mitigate the effects of automation bias. The training should reoc\u00ad\ncur regularly to ensure it is up to date with the system and to ensure the system is used appropriately. Assess\u00ad\nment should be ongoing to ensure that the use of the system with human involvement provides for appropri\u00ad\nate results, i.e., that the involvement of people does not invalidate the system's assessment as safe and effective \nor lead to algorithmic discrimination.", "094c20fa-14b1-497b-b40e-5b99c32cf2fc": "(https://www.ftc.gov/legal-library/browse/cases-proceedings/192-3172-everalbum-inc-matter), and\nagainst Weight Watchers and their subsidiary Kurbo\n(https://www.ftc.gov/legal-library/browse/cases-proceedings/1923228-weight-watchersww)\n69. See, e.g., HIPAA, Pub. L 104-191 (1996); Fair Debt Collection Practices Act (FDCPA), Pub. L. 95-109\n(1977); Family Educational Rights and Privacy Act (FERPA) (20 U.S.C. \u00a7 1232g), Children's Online\nPrivacy Protection Act of 1998, 15 U.S.C. 6501\u20136505, and Confidential Information Protection and\nStatistical Efficiency Act (CIPSEA) (116 Stat. 2899)\n70. Marshall Allen. You Snooze, You Lose: Insurers Make The Old Adage Literally True. ProPublica. Nov.\n21, 2018.\nhttps://www.propublica.org/article/you-snooze-you-lose-insurers-make-the-old-adage-literally-true\n71. Charles Duhigg. How Companies Learn Your Secrets. The New York Times. Feb. 16, 2012.\nhttps://www.nytimes.com/2012/02/19/magazine/shopping-habits.html", "f33bc6b2-858a-46bd-ba56-b6410ce7b11b": "Security \nMP-5.1-002 \nIdentify potential content provenance harms of GAI, such as misinformation or \ndisinformation, deepfakes, including NCII, or tampered content. Enumerate and \nrank risks based on their likelihood and potential impact, and determine how well \nprovenance solutions address speci\ufb01c risks and/or harms. \nInformation Integrity; Dangerous, \nViolent, or Hateful Content; \nObscene, Degrading, and/or \nAbusive Content \nMP-5.1-003 \nConsider disclosing use of GAI to end users in relevant contexts, while considering \nthe objective of disclosure, the context of use, the likelihood and magnitude of the \nrisk posed, the audience of the disclosure, as well as the frequency of the \ndisclosures. \nHuman-AI Con\ufb01guration \nMP-5.1-004 Prioritize GAI structured public feedback processes based on risk assessment \nestimates. \nInformation Integrity; CBRN \nInformation or Capabilities; \nDangerous, Violent, or Hateful \nContent; Harmful Bias and \nHomogenization", "ea01c2f2-4936-4233-8845-855c033c5a09": "\u00ad\u00ad\u00ad\u00ad\u00ad\u00ad\u00ad\nALGORITHMIC DISCRIMINATION Protections\nYou should not face discrimination by algorithms \nand systems should be used and designed in an \nequitable \nway. \nAlgorithmic \ndiscrimination \noccurs when \nautomated systems contribute to unjustified different treatment or \nimpacts disfavoring people based on their race, color, ethnicity, \nsex \n(including \npregnancy, \nchildbirth, \nand \nrelated \nmedical \nconditions, \ngender \nidentity, \nintersex \nstatus, \nand \nsexual \norientation), religion, age, national origin, disability, veteran status, \ngenetic infor-mation, or any other classification protected by law. \nDepending on the specific circumstances, such algorithmic \ndiscrimination may violate legal protections. Designers, developers, \nand deployers of automated systems should take proactive and \ncontinuous measures to protect individuals and communities \nfrom algorithmic discrimination and to use and design systems in \nan equitable way.  This protection should include proactive equity", "641dd569-3b6d-49b4-ab74-5b743949ed5d": "requirements on drivers, such as slowing down near schools or playgrounds.16\nFrom large companies to start-ups, industry is providing innovative solutions that allow \norganizations to mitigate risks to the safety and efficacy of AI systems, both before \ndeployment and through monitoring over time.17 These innovative solutions include risk \nassessments, auditing mechanisms, assessment of organizational procedures, dashboards to allow for ongoing \nmonitoring, documentation procedures specific to model assessments, and many other strategies that aim to \nmitigate risks posed by the use of AI to companies\u2019 reputation, legal responsibilities, and other product safety \nand effectiveness concerns. \nThe Office of Management and Budget (OMB) has called for an expansion of opportunities \nfor meaningful stakeholder engagement in the design of programs and services. OMB also \npoints to numerous examples of effective and proactive stakeholder engagement, including the Community-", "ea99d79c-dacc-4993-a145-2146a1469e05": "SAFE AND EFFECTIVE \nSYSTEMS \nHOW THESE PRINCIPLES CAN MOVE INTO PRACTICE\nReal-life examples of how these principles can become reality, through laws, policies, and practical \ntechnical and sociotechnical approaches to protecting rights, opportunities, and access. \u00ad\u00ad\nExecutive Order 13960 on Promoting the Use of Trustworthy Artificial Intelligence in the \nFederal Government requires that certain federal agencies adhere to nine principles when \ndesigning, developing, acquiring, or using AI for purposes other than national security or \ndefense. These principles\u2014while taking into account the sensitive law enforcement and other contexts in which \nthe federal government may use AI, as opposed to private sector use of AI\u2014require that AI is: (a) lawful and \nrespectful of our Nation\u2019s values; (b) purposeful and performance-driven; (c) accurate, reliable, and effective; (d)", "e8a4ecfe-f6e5-4984-8f0c-694996adfb03": "systems. \n8. Information Integrity: Lowered barrier to entry to generate and support the exchange and \nconsumption of content which may not distinguish fact from opinion or \ufb01ction or acknowledge \nuncertainties, or could be leveraged for large-scale dis- and mis-information campaigns. \n9. Information Security: Lowered barriers for o\ufb00ensive cyber capabilities, including via automated \ndiscovery and exploitation of vulnerabilities to ease hacking, malware, phishing, o\ufb00ensive cyber \n \n \n6 Some commenters have noted that the terms \u201challucination\u201d and \u201cfabrication\u201d anthropomorphize GAI, which \nitself is a risk related to GAI systems as it can inappropriately attribute human characteristics to non-human \nentities.  \n7 What is categorized as sensitive data or sensitive PII can be highly contextual based on the nature of the \ninformation, but examples of sensitive information include information that relates to an information subject\u2019s", "a7b25bc5-d04c-4ce5-b11d-18080ed7322b": "9 Confabulations of falsehoods are most commonly a problem for text-based outputs; for audio, image, or video \ncontent, creative generation of non-factual content can be a desired behavior.  \n10 For example, legal confabulations have been shown to be pervasive in current state-of-the-art LLMs. See also, \ne.g.,", "0422346b-f47b-48ad-890e-93045e292363": "this document as well as in Executive Order on Advancing Racial Equity and Support for Underserved\nCommunities Through the Federal Government:\nhttps://www.whitehouse.gov/briefing-room/presidential-actions/2021/01/20/executive-order\u00ad\nadvancing-racial-equity-and-support-for-underserved-communities-through-the-federal-government/\n106. HealthCare.gov. Navigator - HealthCare.gov Glossary. Accessed May 2, 2022.\nhttps://www.healthcare.gov/glossary/navigator/\n72", "d444272b-84db-47b2-8e39-d070bef54d11": "ENDNOTES\n57. ISO Technical Management Board. ISO/IEC Guide 71:2014. Guide for addressing accessibility in\nstandards. International Standards Organization. 2021. https://www.iso.org/standard/57385.html\n58. World Wide Web Consortium. Web Content Accessibility Guidelines (WCAG) 2.0. Dec. 11, 2008.\nhttps://www.w3.org/TR/WCAG20/\n59. Reva Schwartz, Apostol Vassilev, Kristen Greene, Lori Perine, and Andrew Bert. NIST Special\nPublication 1270: Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. The\nNational Institute of Standards and Technology. March, 2022. https://nvlpubs.nist.gov/nistpubs/\nSpecialPublications/NIST.SP.1270.pdf\n60. See, e.g., the 2014 Federal Trade Commission report \u201cData Brokers A Call for Transparency and\nAccountability\u201d. https://www.ftc.gov/system/files/documents/reports/data-brokers-call-transparency\u00ad\naccountability-report-federal-trade-commission-may-2014/140527databrokerreport.pdf", "84e5065a-6f26-49c3-aeb8-31a8102a856b": "DATA PRIVACY \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nProtect the public from unchecked surveillance \nHeightened oversight of surveillance. Surveillance or monitoring systems should be subject to \nheightened oversight that includes at a minimum assessment of potential harms during design (before deploy\u00ad\nment) and in an ongoing manner, to ensure that the American public\u2019s rights, opportunities, and access are \nprotected. This assessment should be done before deployment and should give special attention to ensure \nthere is not algorithmic discrimination, especially based on community membership, when deployed in a \nspecific real-world context. Such assessment should then be reaffirmed in an ongoing manner as long as the \nsystem is in use.", "3976a13c-4484-47bc-8b1d-0fcb75a19b95": "HUMAN ALTERNATIVES, \nCONSIDERATION, AND \nFALLBACK \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nEquitable. Consideration should be given to ensuring outcomes of the fallback and escalation system are \nequitable when compared to those of the automated system and such that the fallback and escalation \nsystem provides equitable access to underserved communities.105 \nTimely. Human consideration and fallback are only useful if they are conducted and concluded in a \ntimely manner. The determination of what is timely should be made relative to the specific automated \nsystem, and the review system should be staffed and regularly assessed to ensure it is providing timely \nconsideration and fallback. In time-critical systems, this mechanism should be immediately available or,", "88018024-6cf6-4719-ad61-61f79483bb74": "DATA PRIVACY \nEXTRA PROTECTIONS FOR DATA RELATED TO SENSITIVE\nDOMAINS\nSome domains, including health, employment, education, criminal justice, and personal finance, have long been \nsingled out as sensitive domains deserving of enhanced data protections. This is due to the intimate nature of these \ndomains as well as the inability of individuals to opt out of these domains in any meaningful way, and the \nhistorical discrimination that has often accompanied data knowledge.69 Domains understood by the public to be \nsensitive also change over time, including because of technological developments. Tracking and monitoring \ntechnologies, personal tracking devices, and our extensive data footprints are used and misused more than ever \nbefore; as such, the protections afforded by current legal guidelines may be inadequate. The American public \ndeserves assurances that data related to such sensitive domains is protected and used appropriately and only in", "641be3b7-f879-4cc0-bc16-d9cb27069618": "36 \nMEASURE 2.11: Fairness and bias \u2013 as identi\ufb01ed in the MAP function \u2013 are evaluated and results are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.11-001 \nApply use-case appropriate benchmarks (e.g., Bias Benchmark Questions, Real \nHateful or Harmful Prompts, Winogender Schemas15) to quantify systemic bias, \nstereotyping, denigration, and hateful content in GAI system outputs; \nDocument assumptions and limitations of benchmarks, including any actual or \npossible training/test data cross contamination, relative to in-context \ndeployment environment. \nHarmful Bias and Homogenization \nMS-2.11-002 \nConduct fairness assessments to measure systemic bias. Measure GAI system \nperformance across demographic groups and subgroups, addressing both \nquality of service and any allocation of services and resources. Quantify harms \nusing: \ufb01eld testing with sub-group populations to determine likelihood of \nexposure to generated content exhibiting harmful bias, AI red-teaming with", "f12b5467-1c94-4938-98a8-5e0e4e6fff77": "than an applicant who did not attend an HBCU. This was found to be true even when controlling for\nother credit-related factors.32\n\u2022\nA hiring tool that learned the features of a company's employees (predominantly men) rejected women appli\u00ad\ncants for spurious and discriminatory reasons; resumes with the word \u201cwomen\u2019s,\u201d such as \u201cwomen\u2019s\nchess club captain,\u201d were penalized in the candidate ranking.33\n\u2022\nA predictive model marketed as being able to predict whether students are likely to drop out of school was\nused by more than 500 universities across the country. The model was found to use race directly as a predictor,\nand also shown to have large disparities by race; Black students were as many as four times as likely as their\notherwise similar white peers to be deemed at high risk of dropping out. These risk scores are used by advisors \nto guide students towards or away from majors, and some worry that they are being used to guide\nBlack students away from math and science subjects.34\n\u2022", "380caf5a-f592-4a9d-8e55-905836b69ded": "zucked-users-say-they-get-blocked-racism-discussion/2859593002/\n8. See, e.g., Michael Levitt. AirTags are being used to track people and cars. Here's what is being done about it.\nNPR. Feb. 18, 2022. https://www.npr.org/2022/02/18/1080944193/apple-airtags-theft-stalking-privacy-tech;\nSamantha Cole. Police Records Show Women Are Being Stalked With Apple AirTags Across the Country.\nMotherboard. Apr. 6, 2022. https://www.vice.com/en/article/y3vj3y/apple-airtags-police-reports-stalking\u00ad\nharassment\n9. Kristian Lum and William Isaac. To Predict and Serve? Significance. Vol. 13, No. 5, p. 14-19. Oct. 7, 2016.\nhttps://rss.onlinelibrary.wiley.com/doi/full/10.1111/j.1740-9713.2016.00960.x; Aaron Sankin, Dhruv Mehrotra,\nSurya Mattu, and Annie Gilbertson. Crime Prediction Software Promised to Be Free of Biases. New Data Shows\nIt Perpetuates Them. The Markup and Gizmodo. Dec. 2, 2021. https://themarkup.org/prediction\u00ad\nbias/2021/12/02/crime-prediction-software-promised-to-be-free-of-biases-new-data-shows-it-perpetuates\u00ad", "5b9ba636-3418-4270-a189-27f4e5b95ae0": "GOVERN 1.6: Mechanisms are in place to inventory AI systems and are resourced according to organizational risk priorities. \nAction ID \nSuggested Action \nGAI Risks \nGV-1.6-001 Enumerate organizational GAI systems for incorporation into AI system inventory \nand adjust AI system inventory requirements to account for GAI risks. \nInformation Security \nGV-1.6-002 De\ufb01ne any inventory exemptions in organizational policies for GAI systems \nembedded into application software. \nValue Chain and Component \nIntegration \nGV-1.6-003 \nIn addition to general model, governance, and risk information, consider the \nfollowing items in GAI system inventory entries: Data provenance information \n(e.g., source, signatures, versioning, watermarks); Known issues reported from \ninternal bug tracking or external information sharing resources (e.g., AI incident \ndatabase, AVID, CVE, NVD, or OECD AI incident monitor); Human oversight roles \nand responsibilities; Special rights and considerations for intellectual property,", "c3f7bcbe-0afe-4e8b-a6c2-8266ee6bec0a": "WHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nDemonstrate that the system protects against algorithmic discrimination \nIndependent evaluation. As described in the section on Safe and Effective Systems, entities should allow \nindependent evaluation of potential algorithmic discrimination caused by automated systems they use or \noversee. In the case of public sector uses, these independent evaluations should be made public unless law \nenforcement or national security restrictions prevent doing so. Care should be taken to balance individual \nprivacy with evaluation data access needs; in many cases, policy-based and/or technological innovations and \ncontrols allow access to such data without compromising privacy.", "f78abfc0-dc1b-4904-b10f-45b2d75bdffa": "24 \nMAP 2.1: The speci\ufb01c tasks and methods used to implement the tasks that the AI system will support are de\ufb01ned (e.g., classi\ufb01ers, \ngenerative models, recommenders). \nAction ID \nSuggested Action \nGAI Risks \nMP-2.1-001 \nEstablish known assumptions and practices for determining data origin and \ncontent lineage, for documentation and evaluation purposes. \nInformation Integrity \nMP-2.1-002 \nInstitute test and evaluation for data and content \ufb02ows within the GAI system, \nincluding but not limited to, original data sources, data transformations, and \ndecision-making criteria. \nIntellectual Property; Data Privacy \nAI Actor Tasks: TEVV \n \nMAP 2.2: Information about the AI system\u2019s knowledge limits and how system output may be utilized and overseen by humans is \ndocumented. Documentation provides su\ufb03cient information to assist relevant AI Actors when making decisions and taking \nsubsequent actions. \nAction ID \nSuggested Action \nGAI Risks \nMP-2.2-001", "e88db2aa-0248-4c41-9ff5-f64b062d93ad": "NOTICE & \nEXPLANATION \nWHAT SHOULD BE EXPECTED OF AUTOMATED SYSTEMS\nThe expectations for automated systems are meant to serve as a blueprint for the development of additional \ntechnical standards and practices that are tailored for particular sectors and contexts. \nTailored to the level of risk. An assessment should be done to determine the level of risk of the auto\u00ad\nmated system. In settings where the consequences are high as determined by a risk assessment, or extensive \noversight is expected (e.g., in criminal justice or some public sector settings), explanatory mechanisms should \nbe built into the system design so that the system\u2019s full behavior can be explained in advance (i.e., only fully \ntransparent models should be used), rather than as an after-the-decision interpretation. In other settings, the \nextent of explanation provided should be tailored to the risk level. \nValid. The explanation provided by a system should accurately reflect the factors and the influences that led", "481dbfa9-e17c-4a32-bfda-547eb5403563": "State of AI Incident Tracking and Disclosure \nFormal channels do not currently exist to report and document AI incidents. However, a number of \npublicly available databases have been created to document their occurrence. These reporting channels \nmake decisions on an ad hoc basis about what kinds of incidents to track. Some, for example, track by \namount of media coverage.", "60edd255-562c-403c-b6b1-20d1d828e53f": "NIST Trustworthy and Responsible AI  \nNIST AI 600-1 \nArtificial Intelligence Risk Management \nFramework: Generative Artificial \nIntelligence Profile \n \n \n \nThis publication is available free of charge from: \nhttps://doi.org/10.6028/NIST.AI.600-1 \n \nJuly 2024 \n \n \n \n \nU.S. Department of Commerce  \nGina M. Raimondo, Secretary \nNational Institute of Standards and Technology  \nLaurie E. Locascio, NIST Director and Under Secretary of Commerce for Standards and Technology", "810d4e10-aa6e-4399-aee2-0740c4dc03c4": "48 \n\u2022 Data protection \n\u2022 Data retention  \n\u2022 Consistency in use of de\ufb01ning key terms \n\u2022 Decommissioning \n\u2022 Discouraging anonymous use \n\u2022 Education  \n\u2022 Impact assessments  \n\u2022 Incident response \n\u2022 Monitoring \n\u2022 Opt-outs  \n\u2022 Risk-based controls \n\u2022 Risk mapping and measurement \n\u2022 Science-backed TEVV practices \n\u2022 Secure software development practices \n\u2022 Stakeholder engagement \n\u2022 Synthetic content detection and \nlabeling tools and techniques \n\u2022 Whistleblower protections \n\u2022 Workforce diversity and \ninterdisciplinary teams\nEstablishing acceptable use policies and guidance for the use of GAI in formal human-AI teaming settings \nas well as di\ufb00erent levels of human-AI con\ufb01gurations can help to decrease risks arising from misuse, \nabuse, inappropriate repurpose, and misalignment between systems and users. These practices are just \none example of adapting existing governance protocols for GAI contexts.  \nA.1.3. Third-Party Considerations"}}

Tasks/Task 4/test_questions.json

@@ -0,0 +1 @@
+{"a188ff31-68db-46bc-b419-ac0fdecc6b1b": "What is data poisoning in the context of GAI and how does it affect model outputs?", "c9f79a82-8775-4461-ac2f-b7b4e8734e96": "How do GAI systems pose intellectual property risks related to copyrighted material in their training data?", "80b370f2-6d21-4ff6-bc9b-763317af5321": "How can governance structures help mitigate bias in human-based systems?", "d4e20011-f4f9-4de6-8453-8144d77ba3a7": "What are some common concerns associated with human-based systems in terms of bias and effectiveness?", "12534c5f-eb68-41cb-a879-b908e95a65d0": "What are the best practices for establishing policies for data collection and retention to mitigate risks such as data privacy breaches and harmful biases?", "11951856-b35d-4123-ab15-80c38cd82195": "How can organizations effectively implement policies to protect third-party intellectual property and training data?", "869c7e52-2b77-4144-af5f-d5beebc138f9": "What are the key safeguards that should be included in automated systems to protect the public from harm?", "a310a614-4bd4-4302-ab42-37b6faa7d838": "How can early-stage public consultation improve the safety and effectiveness of automated systems?", "468bff4b-cdbb-489a-9cbd-95d9062f93dc": "What are some of the key investigative projects that Surya Mattu has worked on at The Markup?", "3e5b4545-4ceb-424f-b519-16c3870dd543": "How has Mariah Montgomery's role as National Campaign Director at the Partnership for Working Families impacted labor rights and policies?", "0f0eea1d-f05e-458e-b31a-e955629c7e44": "How can organizations effectively integrate pre- and post-deployment feedback into their monitoring processes for GAI models?", "6cc0bcc5-9c82-49a6-b118-3e333e70ee9e": "What are the benefits of using AI red-teaming in the pre-deployment testing phase for capturing external feedback?", "4441faa1-8f27-4fc7-bea0-847caa1c1505": "What are the potential negative impacts of automated systems on individuals and communities?", "0db8fdee-99cf-47c6-9d8d-a85f3b294826": "How can confirmation bias affect the effectiveness of safety mechanisms in technology?", "57db460e-0123-4edf-b7df-87a967a60c26": "What are the key safety metrics used to evaluate AI system reliability and robustness?", "48589831-4f3c-4bf6-9cb4-bc4277c489dd": "How can AI systems be designed to fail safely when operating beyond their knowledge limits?", "1df11168-7aa5-4b43-91df-c14c32f01440": "What are the risks associated with data brokers collecting consumer data without permission?", "2127b35f-68cd-4e5f-a669-a6a4bb532fa8": "How does the use of surveillance technologies in schools and workplaces impact mental health?", "afefb290-48ec-450c-b530-5fe1b6c5340b": "What is ballot curing and how does it impact the election process?", "eecbf085-2f16-45c4-ba65-35813ca84568": "How do different states handle signature discrepancies in mail-in ballots?", "43b6add5-244e-4c11-be3b-0944fecfa6b9": "What are the best practices for detecting and mitigating algorithmic bias according to the Brookings Report?", "37bbd6b6-d24a-4b73-a4f4-f532d8c1793a": "How can public agencies implement Algorithmic Impact Assessments to ensure accountability, as suggested by the AI Now Institute Report?", "318fe73a-0591-41e8-b65e-925c71b2caab": "How is the federal government addressing discrimination in mortgage lending through the Department of Justice's nationwide initiative?", "56664bc2-0933-4e58-8d03-5c06b9d06c04": "What role do federal agencies like the Consumer Financial Protection Bureau play in the Action Plan to Advance Property Appraisal and Valuation Equity?", "7f8b418c-6e85-4ab0-83db-b7ed7dc49a45": "What are the best practices for updating due diligence processes to include intellectual property and data privacy for GAI acquisitions?", "e81617a3-9609-4012-ba46-caa374c306de": "How can organizations effectively monitor and assess third-party GAI risks in real-time?", "054e5797-d024-41bd-8de9-983d038a8797": "What are the best practices for performing disparity testing and making the results public?", "fdb81ad2-acf2-4aa4-b551-fe766d22f273": "How can organizations effectively mitigate disparities identified through testing?", "09a4ef32-a01e-4ca9-9bf6-4704e328ccef": "How can people protect themselves from being tracked by devices originally meant for finding lost items?", "e314f460-f6e2-4d11-b612-d51529a9dee6": "What are the potential issues with using algorithms to deploy police in neighborhoods?", "741f5989-422f-4bc5-9f72-0f3b22bb4f25": "What are the mental health impacts of NCII on women and sexual minorities?", "19592c9a-0621-4629-bdfc-8a08f0d396b4": "How can GAI training datasets be protected from including CSAM and NCII?", "f95100da-f55f-4402-909d-fdde5cf17d25": "What are the key challenges in designing non-discriminatory AI technology discussed in the panel?", "bb3e7970-5b1e-4e98-87ad-b30d33ff6a89": "How can community participation enhance human-computer interaction in AI systems?", "796ffa10-1532-4fa1-b832-d8ee058d410d": "What are the potential sociotechnical harms of algorithmic systems as discussed by Shelby et al (2023)?", "2c38117e-4b2d-4553-b319-f4ba3997996e": "How does training on generated data affect AI models according to Shumailov et al (2023)?", "3b9c9379-cc75-4b9d-a68a-dc6b0a48fd9c": "What are the key suggested actions for managing GAI risks according to the AI RMF 10 and Playbook?", "1a02235f-7bf0-4e7e-8149-ab610eacb769": "How do the suggested actions for managing GAI risks vary depending on the stage of the GAI lifecycle?", "08cbf993-d60b-4982-bf84-140c29d30450": "How can organizations ensure that consent practices do not allow for abusive surveillance practices?", "e253b5ac-feb7-4116-9e68-d2c817da36a5": "What are the best practices for re-acquiring consent if the use case of data changes or if data is transferred to another entity?", "0260750e-4f7d-4c1b-b0b7-ae4c36cc8fc3": "What are the key principles outlined in the AI Bill of Rights?", "39863570-2d41-4d21-bde1-1afc78c157b0": "How does the AI Bill of Rights address algorithmic discrimination?", "8b4fd9d7-e1d4-472e-bd34-35fa98299c07": "How can we effectively track and document instances of anthropomorphization in GAI system interfaces?", "1f34befe-4432-419f-8465-066a0d82ff77": "What are the best practices for verifying the provenance of GAI system training data and TEVV data?", "3e35db8c-c1b3-4b9f-b6c0-a3fd6e52d2b0": "What is the importance of having demographically and interdisciplinarily diverse AI red teams in pre-deployment contexts?", "0c3b35ca-f421-41e4-b016-8f367561acbe": "How can general public involvement in AI red-teaming contribute to identifying flaws in AI models?", "d73655e4-93f0-41c5-b69e-814ff8189db8": "QUESTION #1: How are major universities using race as a predictor of student success?", "bb8b9729-d1b6-407d-9f0c-aa1bd62a8d78": "QUESTION #2: What concerns do students and professors have about using race as a predictor in education?", "3f1dec42-4087-4e06-9e7e-491c96cdee67": "How can AI-enabled systems contribute to building better and more innovative infrastructure?", "9a94cfb2-25b9-4aa8-94c5-987c53fa42bf": "What lessons from urban planning can be applied to the integration of AI technologies in communities?", "f445449e-b75f-44ee-a819-018ad630bd35": "What are the benefits of having a human alternative to automated systems?", "58fd202f-0791-411a-9124-09381dbbad11": "How can one ensure timely human consideration and remedy when an automated system fails?", "26ee0f55-a947-440f-b4bc-4b7def4e3545": "What are the main findings of the Department of Justice's report on the risk assessment tool for predicting recidivism?", "dcb01564-a34f-42a7-ac6c-13764525a7d2": "How is the Department of Justice addressing the disparities in the risk assessment tool for predicting recidivism among different groups of color?", "8e29d29a-fc98-4a6f-b42b-580fc084dd71": "What is the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government?", "4bbc6d4b-6b67-4831-8bca-853eb46aec3a": "What were President Biden's remarks on the Supreme Court decision to overturn Roe v Wade?", "919fdd1d-2abb-472e-ac8d-bde9df2bb391": "What are the best practices for re-assessing model risks after implementing fine-tuning or retrieval-augmented generation?", "a795e873-419b-454b-8598-fb0c49a7e5cc": "How can organizations effectively review and manage training data to prevent the reproduction of intellectual property or CBRN information in AI outputs?", "242b750e-1236-41f7-a1cc-eedef8f0427d": "What are some common examples of AI incidents that organizations should be aware of?", "c506e557-776f-42ed-99f9-c752ac2bb94b": "How can organizations effectively track and document the provenance of datasets to identify AI-generated data issues?", "426616e2-6297-47c3-89c7-71ec1186cdba": "What is the role of the American Civil Liberties Union in protecting digital privacy?", "db43af55-434d-441f-8dc7-acc8ff3f8432": "How does the Center for Democracy & Technology advocate for internet freedom and security?", "f2913868-28a6-4558-904a-0486fbfc1f6e": "How can organizations ensure the accuracy of predictions or recommendations generated by automated systems?", "5a4faa70-0364-4fd0-9c98-b26fb63f7786": "What are the best practices for implementing ongoing monitoring procedures for automated systems?", "3ad57490-e4f7-4fd2-bff4-93211043ec13": "What are the key considerations for implementing automated systems in sensitive domains like criminal justice and health?", "3a05d7ba-2e46-406b-aeb9-51b33efff15f": "How can organizations ensure meaningful oversight and human consideration in high-risk automated decision-making systems?", "6ae09ea8-3090-401b-9f1e-4ce5270152cd": "What are the privacy expectations for automated systems handling sensitive data?", "207207ff-faab-4342-b76f-ef0c6fac88c9": "How should consent be managed for automated systems collecting sensitive data?", "d15a10aa-36cb-4f3a-9f9e-2c0416ce1084": "What is the contact information for inquiries related to NIST AI publications?", "c9f4fb11-9365-4354-aa94-7cc93efcafb5": "Where can I find additional information about NIST AI publications?", "4aebac20-11d4-42c8-be6a-f7ac4e43cbbc": "How can organizations effectively combat automation bias in automated systems?", "71537f88-7e77-4720-9cc0-bca516b4721f": "What are the best practices for training individuals to properly interpret outputs from automated systems?", "e004e796-65d5-4109-89bd-472cae5b6c75": "What were the FTC's findings in the case against Everalbum Inc?", "290cd0b2-456b-41bc-bf0e-3ea3e32f480d": "How did the FTC address privacy concerns in the case against Weight Watchers and Kurbo?", "1c416614-5e28-45f4-9e8e-937971dcff9a": "What are the potential harms of GAI related to misinformation, disinformation, and deepfakes?", "d83ab93d-9be0-488d-94fd-8e58074a3388": "How should organizations disclose the use of GAI to end users to mitigate risks?", "bfc45e93-d073-4348-8fb1-03dfaf4e73f3": "What measures can designers and developers take to prevent algorithmic discrimination?", "4819bdb4-1724-4318-855c-9c4f680c0655": "How does algorithmic discrimination impact different protected classes such as race, gender, and disability?", "a8a96840-d387-42d9-9b56-f05b73027f5c": "What are some innovative solutions provided by the industry to mitigate risks to the safety and efficacy of AI systems?", "7fdb6c15-c3f8-4327-b2fe-0169c08ce375": "How does the Office of Management and Budget (OMB) suggest expanding opportunities for stakeholder engagement in program design?", "3509c40f-7af0-49a5-bd16-c7da584b3980": "What are the nine principles outlined in Executive Order 13960 for the use of AI in the federal government?", "a86eba64-72a8-4afa-a7f5-8c50c3b0c660": "How can laws and policies ensure that AI systems are accurate, reliable, and effective in real-life applications?", "9eee9d68-6e0f-4430-989f-cb569677d74c": "How can we distinguish between fact and opinion in the content generated by AI systems?", "6fba0797-2aaa-4686-9325-999b5396f47b": "What are the risks associated with the anthropomorphization of General AI (GAI) systems?", "449ab90b-3762-4d3e-99ea-899bd340c42b": "What are confabulations in the context of text-based outputs?", "1c57be24-8e1d-4a3a-a29e-1d153c019510": "How do legal confabulations manifest in state-of-the-art language models?", "6b30e12e-cecf-4cd7-936e-84468c950a36": "What is the purpose of the Executive Order on Advancing Racial Equity and Support for Underserved Communities Through the Federal Government?", "5547bf9b-ceae-4386-a486-7708637ab6a1": "What role do Navigators play according to HealthCaregov?", "a520c4cc-f2f6-4dd8-bd3a-a1a750440209": "What are the key principles outlined in the ISO/IEC Guide 71:2014 for addressing accessibility in standards?", "d72c0d17-abee-470b-8725-abf4aad59b3f": "How do the Web Content Accessibility Guidelines (WCAG) 20 impact web development practices?", "8e31e286-3ac3-488f-a211-4575fd663a17": "What are the key expectations for automated systems to ensure data privacy and protection from unchecked surveillance?", "76f71eb0-f3b8-425d-8772-65a5d214634f": "How can heightened oversight of surveillance systems prevent algorithmic discrimination based on community membership?", "eb21dff3-4dd0-47af-a449-b9b525386911": "What are the key considerations for ensuring equitable outcomes in fallback and escalation systems for automated systems?", "645a1801-9128-4977-947d-5437b8933966": "How can organizations ensure that human consideration and fallback mechanisms are conducted in a timely manner for automated systems?", "50321a04-5130-43ab-9305-cc1d548da8e0": "What are the extra protections for data related to sensitive domains like health and personal finance?", "32f6e506-6e82-41f7-b80c-f0702a537ca2": "How do technological developments impact the sensitivity of data domains and the need for enhanced data protections?", "7a80ac97-319d-452b-a900-e739da72ab44": "What are some benchmarks used to quantify systemic bias in GAI system outputs?", "bedb600e-b951-4e89-9442-24b971ff1b21": "How can fairness assessments help measure systemic bias in GAI systems?", "5fa67d29-3be8-4c81-a7c9-1a4d5dfa0ba7": "What are the potential biases in hiring tools that learn from a company's predominantly male employee base?", "868637a7-88fa-4891-bcfd-da1d37772744": "How do predictive models that use race as a factor affect Black students' academic guidance and major selection?", "6410524d-24f8-4aaf-8b70-5dcfc8272cd0": "What measures are being taken to prevent the misuse of Apple AirTags for stalking and harassment?", "d0990582-29f1-41c2-90e1-89c4efc58153": "How does crime prediction software perpetuate biases despite promises of being free from them?", "cbc06d96-6605-45f4-8067-0342ab04aac4": "What are the key elements to consider when incorporating GAI systems into an AI system inventory?", "b16b4f6b-0ec2-49bc-9453-9bbf1a8feea5": "How should organizations handle inventory exemptions for GAI systems embedded into application software?", "b1da9e4e-62f7-4d08-ac87-2b196fa9114e": "What measures can be taken to ensure automated systems protect against algorithmic discrimination?", "aa279423-ea2b-4fa2-beb1-7a6e1400c36f": "How can independent evaluations of automated systems be conducted without compromising individual privacy?", "8a9ae766-2f74-4272-bd84-e95787e5e943": "What are the best practices for determining data origin and content lineage in AI systems?", "2567081e-89ba-4d98-a746-eaf8503e5c5d": "How can test and evaluation processes be instituted for data and content flows within an AI system?", "fa585f44-6fb6-443b-983e-6304d9c2f5e1": "What are the expectations for automated systems in high-risk settings like criminal justice?", "f51b4b1e-689f-4a47-82a2-d9a9a0d30ab7": "How should the level of risk influence the design of explanatory mechanisms in automated systems?", "ecb05eb6-335e-4451-bf2e-4c8ad8e800bf": "What are the current methods for reporting AI incidents?", "51d512f0-8849-48de-a188-5aab8ddee724": "How do publicly available databases decide which AI incidents to track?", "80d2e492-0668-4e82-b83e-d1cef2355444": "What is the NIST AI 600-1 framework about?", "b7e8353e-ffb6-41a5-a321-d6b5521a03d5": "How does the NIST Trustworthy and Responsible AI framework address generative artificial intelligence risks?", "acbfd37b-65e1-440b-b4b1-9b3ee9a15fac": "What are the best practices for establishing acceptable use policies for GAI in human-AI teaming settings?", "c86085f1-bc71-4d66-8869-d5335b328ec7": "How can organizations effectively implement synthetic content detection and labeling tools?"}

Tasks/Task 4/val_dataset (2).jsonl

@@ -0,0 +1 @@
+{"questions": {"61c5d3ec-11af-4a36-a028-e9e22afb5a8f": "What are the five principles outlined in the Blueprint for an AI Bill of Rights?", "662911a9-7407-4a24-95ff-2350dde354be": "How can communities and industry implement the practices suggested in the Blueprint for an AI Bill of Rights?", "09ded63e-b364-42e7-9677-e1dfa4932b9b": "What are the best practices for providing independent evaluators access to automated systems while ensuring privacy and security?", "6b539180-33d5-4cd2-abf5-63cbe6178e6a": "How can organizations ensure that evaluator access to automated systems remains truly independent and cannot be revoked without reasonable justification?", "f35c9f92-67ac-4772-9dab-6cf2ae32812f": "What are the legal requirements for providing notice when making a video recording of someone?", "d0edffef-580d-4890-a6dd-e08925fadd27": "How are companies and researchers improving automated systems to explain decisions that impact consumers?", "4a36f5cd-0f9d-42ba-bd8e-d0eaf0af2d52": "How do advertisement delivery systems reinforce racial and gender stereotypes?", "1c6de01d-b59d-4421-9339-0e501b4fd2b9": "What are the issues faced by transgender travelers with TSA body scanners at airport checkpoints?", "155db437-082c-44f4-8751-960146c3512c": "What are the five principles outlined in the Blueprint for an AI Bill of Rights?", "95cae333-a114-41e8-98f5-10619377f6bf": "How can organizations apply the Blueprint for an AI Bill of Rights to protect civil rights and privacy?", "077e8ee5-5768-4967-b8ed-891c6cc0085d": "What are the benefits of having a human fallback mechanism in automated systems?", "8edf6c51-407d-478c-832a-ef103ea3709e": "How do automated signature matching systems impact voters with mental or physical disabilities?", "7058b177-27f4-4d6b-a478-176ead46f325": "What are the best practices for documenting the sources and types of training data in AI models?", "1e48abdd-a664-4c7a-8f19-151ca61e5006": "How can user feedback be effectively integrated into system updates to address problematic content?", "e5aba341-abc2-4965-a224-fa10823f4d2f": "What is the two-part test used in the AI Bill of Rights framework to determine which systems are in scope?", "23c3711f-c55b-49e5-9936-22d6bfc010af": "How does the AI Bill of Rights framework ensure that automated systems do not negatively impact the American public's rights and access to critical resources?", "08a12dd0-5dd7-4f87-8913-d86a9cc2c8b7": "What are adversarial role-playing exercises and how do they help in identifying failure modes in GAI systems?", "7d2b3bbe-6d0b-470d-b85d-d0c636ac4354": "How can profiling threats and negative impacts improve the security of GAI systems?", "c385b92d-1c01-48ae-be4c-f6b42b5e6af6": "What are the potential negative impacts of school surveillance on students via laptops?", "b4286477-40f0-46b8-bba8-4fe204b0dafa": "How does \"Bossware\" affect the health of employees according to the Center for Democracy & Technology report?", "6c98dd15-2a73-4c66-8a6a-c578c67a2434": "How can employers ensure their use of AI in hiring complies with the Americans with Disabilities Act (ADA)?", "00ab3a02-dffb-482b-ad10-3cab6ad77520": "What are the potential risks of using healthcare algorithms that rely on past medical costs to predict future needs?", "510ed741-6a36-4d13-a7dc-6a42262136be": "What are some effective context-based measures to identify new impacts of GAI systems?", "8f74dbe1-c3ed-48ca-9635-d701d26e829a": "How can regular engagements with AI Actors help in evaluating unanticipated impacts of GAI systems?", "3809c393-b89e-494c-b529-c65e601c1544": "What are acceptable use policies for GAI interfaces and how do they determine the types of queries GAI applications should refuse to respond to?", "edb9b7b1-11c1-421c-a07f-7abe3d6e7c21": "How can organizations establish effective user feedback mechanisms for GAI systems, and what should these mechanisms include?", "5a48e740-85f0-48c7-b0c7-6247c384f052": "How often should adversarial testing be conducted to effectively map and measure GAI risks?", "d6567db0-b18c-4dcb-b80c-146f2047bc13": "What are the benefits of evaluating GAI system performance in real-world scenarios compared to controlled testing environments?", "59a37c01-7bac-4f9d-980f-48f5489e61e6": "What are the common statistics reported about who chooses the human alternative in automated systems?", "e24a71f0-8b86-461a-92bd-fa6cef7ca33b": "How often should reports on the accessibility, timeliness, and effectiveness of human consideration and fallback be made public?", "63dcc302-d64d-47f5-a304-a64d4d6642b4": "What are some examples of companies that have successfully implemented bias testing in their product quality assessment?", "9b9b4805-12cb-453d-a3f4-ddbb20679c39": "How are federal government agencies developing standards to prevent algorithmic discrimination?", "7b3d457a-d0bf-4b13-b59c-df184af98f08": "What are some common protections against unlawful surveillance and violations of privacy in both public and private sectors?", "9473baea-32cd-4147-a547-5d45b0daa757": "How can individuals ensure equitable access to education, housing, and employment opportunities?", "d4388801-831e-45e0-bf67-b67974027277": "What are the key principles outlined in the AI Bill of Rights?", "d4107956-2806-4098-a79e-e753cab1bf82": "How can the AI Bill of Rights be practically implemented in technical systems?", "829774bb-4770-46cf-9f1b-86f51e7b6679": "How can you ensure the data used in automated systems is of high quality and relevant to the task?", "79c355b3-3945-402d-9d15-e460689ba635": "What methods can be employed to measure and limit errors from data entry in automated systems?", "e558dbd7-ca81-4070-9777-49636694d674": "What are some reasons why certain risks cannot be measured quantitatively in AI systems?", "e1ce22f6-cad0-4bbe-87ae-5222158a4393": "How can organizations involve independent assessors and domain experts in the regular assessment of AI systems?", "ae84398b-1649-4cce-8fa2-6295c80f7ec9": "What are the risks associated with confabulated content in healthcare applications using GAI?", "648a7032-05c8-45c2-a7bb-2dca8fa9ffd0": "How can confabulated logic or citations from GAI systems mislead users?", "2b743770-5d66-4aa8-b9b4-c33adc78c1e3": "How can companies ethically use data to monitor employee performance without violating privacy?", "3b6f61ff-349d-4817-8c82-d064b9a71c86": "What are the legal implications of employers using surveillance data to intervene in employee discussions?", "f596fded-c16b-49cb-b400-734c65b185af": "What are the risks of using AI in high-stakes settings as highlighted by Pamela Wisniewski and Seny Kamara?", "1626655d-7f72-4d0a-9170-3abdc8ed86ec": "Why is it important to place trust in people rather than technologies when designing AI systems?", "cec6f35c-1b45-4d56-8c2f-aef7bc860a01": "How can organizations ensure that their demographic assessments are inclusive of all protected classifications?", "12aca964-2112-4b36-8a40-14ab1512ac75": "What are the best practices for separating demographic data used for disparity assessment from data used in automated systems?", "53a48063-f4fb-482f-bd70-36915ec63956": "What are some emerging technologies being used to improve social welfare systems?", "7fdbbfed-73aa-45a8-9f1c-58ec2c0f3912": "How can digital welfare systems impact life chances according to experts like Christiaan van Veen?", "0ed0fb9c-47c4-4c7c-a5ae-d7e3a35670a1": "What are some best practices for developers to ensure privacy by design in smartphone apps?", "88297ffa-b5ca-460c-81ed-a61975ab39ef": "How can developers make app permissions clear and use-specific for users?", "38409d77-4936-4266-a7f3-2d910d3bea91": "What are the privacy implications of using biometric identification technologies in New York schools?", "3d2d3a9e-a6a7-49f5-bdd8-5db95fc8b602": "What are the reporting requirements for employers who surveil employees during a labor dispute?", "ca685f83-ccd7-4a17-a31d-bfc648b58840": "What measures are included in the AI Bill of Rights to ensure automated systems are safe and effective?", "ce1fdffd-851d-463e-8f24-4596865b62dc": "How does the AI Bill of Rights propose to handle the risks and potential impacts of automated systems?", "1a82989c-3ead-4aea-9098-53d3dca7f9b7": "What are the potential downstream impacts of errors in third-party GAI components on system accuracy and robustness?", "a30ea710-3349-4357-8dcb-915f6c69f2da": "How can inaccuracies in test dataset labels affect the stability and robustness of GAI benchmarks?", "004b52ee-6a49-47d7-a4bd-77ec96fadc31": "What are the best practices for developing and updating GAI system incident response and recovery plans?", "a5ad1cc1-318a-4210-8838-22015d780344": "How can organizations ensure their response and recovery plans account for the entire GAI system value chain?", "f05e4729-18f1-4664-9f41-2ad997f9d726": "How can we assess the proportion of synthetic to non-synthetic training data in AI models?", "81c90ac3-caf0-4c9d-8e02-8c62d26a047e": "What are the best practices for documenting the environmental impacts of AI model development and deployment?", "0abf12fc-3e73-41e5-8594-5e2bb6ecdb24": "What are the primary considerations for organizations designing and developing GAI according to the GAI PWG consultation process?", "e3abf868-922a-42e7-8c5a-b1ff0a353d39": "How can governance principles and techniques be applied to manage risks in GAI systems?", "55c79cd5-dee3-4e43-b8a3-839028518379": "What are the key considerations for documenting the intended purposes and beneficial uses of an AI system?", "456333eb-689e-4896-b2d4-0cf136672c77": "How do internal vs external use and narrow vs broad application scope impact the identification of intended purposes for AI systems?", "8834b86c-b1b9-43d6-92e0-3c64ca09e854": "How can feedback from internal and external AI actors be used to assess the impact of AI-generated content?", "e84f1a90-e702-4594-84b8-5c5b67352195": "What are the benefits of using real-time auditing tools for tracking and validating the lineage and authenticity of AI-generated data?", "490b6ca7-059f-41fe-82ae-b8d2c3890cf1": "What are the main findings of Carlini et al (2024) regarding the vulnerabilities in production language models?", "59bed72b-bd80-47c3-bb57-08dd086ecf9d": "How does the study by Chandra et al (2023) propose to combat Chinese influence operations and disinformation?", "625e3e66-e1fc-4223-a201-e88b765f449e": "What is the role of the Electronic Privacy Information Center (EPIC) in AI policy and regulation?", "da4a10c9-db2a-45fa-bad5-b66ef842c023": "How does the Innocence Project utilize AI to support its mission?", "40ab1b55-bc53-4cae-8f7e-4657a5b2bdc2": "What is the role of the National Center for Missing & Exploited Children?", "46de7819-7250-4050-8bf9-4635a1a02f3e": "How does the New York Civil Liberties Union contribute to civil rights advocacy?", "6feae899-9900-454f-a64d-39e842af8c76": "How can AI tools be misused in the development of chemical or biological agents?", "36826afc-57e4-4d70-bc7e-4ca62e3e3e67": "What are the potential risks associated with the use of biological design tools (BDTs) in chemistry and biology?", "84440495-e768-4885-b78b-d8a0c17f3809": "How can expert AI red-teamers enhance the effectiveness of general public AI red-teamers?", "9c3a8107-d49c-4dc0-9f78-d71a506df892": "What are the benefits of using GAI-led red-teaming compared to human red-teamers alone?", "068d8bd2-9336-4e18-bd93-2199100e631f": "How can error ranges be calculated and included in explanations for decision-making systems?", "3138ca26-38b8-4e17-9b31-b38bc8a8eb4f": "What are the best practices for balancing usability and interface complexity when presenting decision-making information?", "095919bc-18fa-4316-b1e8-07572983b77b": "What are the potential benefits and drawbacks of using predictive policing in the criminal justice system?", "39406f17-a757-4201-91b5-284ba4ebbd39": "How can data-driven approaches be balanced with the need for community safety in criminal justice reform?", "2744b9cf-981d-42e5-aed3-bb8e5acb0b2e": "What are the reporting expectations for entities developing or using automated systems?", "798b53f4-f798-418a-abcd-6dd05f707c67": "How can the public access the Agency Inventories of AI Use Cases provided by the National Artificial Intelligence Initiative Office?", "d7fa2d65-26f8-4442-86f6-f1d6256e588a": "What are some effective methods for monitoring and assessing high-impact systems in qualitative user experience research?", "e50c31b3-bab1-4064-baa1-199c946d9789": "How can organizations ensure equity standards are maintained in algorithmic systems, and what steps should be taken if these standards are not met?", "644dcaa5-1731-43fe-b0f5-c6a4bc05564e": "What factors should be considered when updating or defining risk tiers for General Artificial Intelligence (GAI)?", "def43eb9-80b0-4ad2-9198-d84ecb89c720": "How can the psychological impacts of GAI, such as anthropomorphization and emotional entanglement, be mitigated?", "8495a23f-4bb7-47ac-8c54-58cf5675cdd7": "What are the best practices for establishing policies to manage risks related to rollover and fallback technologies in GAI systems?", "74ae51e9-63b3-48ce-9be7-4f88052d7bd6": "How can organizations ensure clear assignment of liability and responsibility in vendor contracts for GAI technologies?", "11cdd3ed-e09b-463d-9853-0be811073b75": "What are the best practices for ensuring the confidentiality of AI training data and model weights?", "9ca1ff0e-0cd9-4362-aca9-fd904077c845": "How can potential attack points in AI systems be identified and secured?", "8fe0054d-51ba-48c5-8cc5-259b2b96f535": "How can AI-powered cameras in delivery vans be improved to avoid incorrectly penalizing drivers?", "03b9f17b-0b61-401b-bc65-47d0655f31d8": "What are the common issues faced by companies using AI to monitor road safety habits of drivers?", "6d622041-fccf-4eb4-9a53-f7d7577856f8": "What are the differences in resource usage between AI training and inference?", "a1738003-3e17-48e7-86a2-1410bc0f1c07": "How can we verify the effectiveness of carbon capture programs for AI training?", "d15e0c10-378f-48a3-9a5c-be0c618106b4": "What protocols should be in place to ensure the safe deactivation of AI systems?", "7e7e2c28-ea80-4568-a71a-41966f9f117f": "What factors need to be considered when decommissioning AI systems to prevent data leakage and ensure security?", "57073541-fc8c-43cd-8b42-f9497eb501af": "What are the best practices for limiting access to sensitive data based on necessity and local control?", "92d9e36d-0fef-4b2e-b40d-ff2b800fcf10": "How should organizations report data security lapses or breaches involving sensitive data?", "6f7aa060-c19a-4614-83d2-134828a7e956": "What is the purpose of the email address ai-equity@ostpeopgov created by OSTP?", "6b95bc28-dbb4-408f-8c5b-f5b37073b6fd": "Where can I find the full responses to the OSTP's Request For Information (RFI) on biometric technologies?", "4776eaa1-b6f0-440c-a6be-923bbf49687d": "What are the practical steps to implement ethical principles in technology?", "acf74d86-1184-4092-8a1d-3ca58f5fe97a": "How can risk management be integrated into technological innovation to protect people from harm?", "2c1b02c6-1919-49ea-beff-165567d20b47": "What are the key capabilities needed for automated systems to help users make consent, access, and control decisions in a complex data ecosystem?", "2d15dfed-c66d-4fac-89dd-3aded02ec63e": "How can independent evaluations of data policies help ensure data privacy and user control in automated systems?", "e71beb7c-7564-4f2c-83f7-ec9bb3134847": "How can the rate of implementing recommendations from security checks and incidents be measured effectively?", "6c079fa0-60c3-4c8d-826a-2816c65d3ea0": "What are the best practices for performing AI red-teaming to assess resilience against various types of attacks?", "543e9bfb-b5f4-4247-89c8-41e0e7fb11a9": "What are the legal and regulatory requirements for reporting GAI incidents under HIPAA?", "8613b055-c817-4a59-84cf-1ae29a7c2269": "How does the NHTSA's 2022 autonomous vehicle crash reporting requirements impact AI deployment and monitoring?", "ce252388-c4d9-4968-aadf-218b47f609a5": "How do you document the justification for each data attribute in an automated system?", "f46088d7-1004-41cb-87c5-8a2b0bcdef59": "What are the best practices for ensuring that the use of high-dimensional data attributes does not violate applicable laws?", "b5f49997-5049-4865-9b5b-c18d880e2baf": "How can organizations adjust their governance regimes to effectively manage the risks associated with generative AI systems?", "eeb5acfd-3be2-4488-b45e-e0979bd5c855": "What are the key considerations for third-party governance across the AI value chain when dealing with generative AI?"}, "relevant_contexts": {"61c5d3ec-11af-4a36-a028-e9e22afb5a8f": ["80e81c8c-bb97-4604-bdef-dcc56813587a"], "662911a9-7407-4a24-95ff-2350dde354be": ["80e81c8c-bb97-4604-bdef-dcc56813587a"], "09ded63e-b364-42e7-9677-e1dfa4932b9b": ["d0a6097e-42c8-499f-8d6d-bcfae7f992d5"], "6b539180-33d5-4cd2-abf5-63cbe6178e6a": ["d0a6097e-42c8-499f-8d6d-bcfae7f992d5"], "f35c9f92-67ac-4772-9dab-6cf2ae32812f": ["51421b31-1a41-49da-a2c2-65df54ae93ce"], "d0edffef-580d-4890-a6dd-e08925fadd27": ["51421b31-1a41-49da-a2c2-65df54ae93ce"], "4a36f5cd-0f9d-42ba-bd8e-d0eaf0af2d52": ["758f783b-3fdc-4890-9de4-da3c035c1141"], "1c6de01d-b59d-4421-9339-0e501b4fd2b9": ["758f783b-3fdc-4890-9de4-da3c035c1141"], "155db437-082c-44f4-8751-960146c3512c": ["96838aa0-1bf7-4ae3-a8d7-5d093e9feb39"], "95cae333-a114-41e8-98f5-10619377f6bf": ["96838aa0-1bf7-4ae3-a8d7-5d093e9feb39"], "077e8ee5-5768-4967-b8ed-891c6cc0085d": ["66c96cba-2674-4734-a869-d002faab751c"], "8edf6c51-407d-478c-832a-ef103ea3709e": ["66c96cba-2674-4734-a869-d002faab751c"], "7058b177-27f4-4d6b-a478-176ead46f325": ["2689bb50-4ffd-4610-856c-c8fad4ab7285"], "1e48abdd-a664-4c7a-8f19-151ca61e5006": ["2689bb50-4ffd-4610-856c-c8fad4ab7285"], "e5aba341-abc2-4965-a224-fa10823f4d2f": ["7515dd00-b05d-49ea-baa0-7cedeb05eb39"], "23c3711f-c55b-49e5-9936-22d6bfc010af": ["7515dd00-b05d-49ea-baa0-7cedeb05eb39"], "08a12dd0-5dd7-4f87-8913-d86a9cc2c8b7": ["f339987a-b2cd-4258-85c5-a864712a9e98"], "7d2b3bbe-6d0b-470d-b85d-d0c636ac4354": ["f339987a-b2cd-4258-85c5-a864712a9e98"], "c385b92d-1c01-48ae-be4c-f6b42b5e6af6": ["673465c5-faf7-4ab1-86e0-d7cc5751143d"], "b4286477-40f0-46b8-bba8-4fe204b0dafa": ["673465c5-faf7-4ab1-86e0-d7cc5751143d"], "6c98dd15-2a73-4c66-8a6a-c578c67a2434": ["3df80c8e-fd5b-436c-9411-42e36faeeaef"], "00ab3a02-dffb-482b-ad10-3cab6ad77520": ["3df80c8e-fd5b-436c-9411-42e36faeeaef"], "510ed741-6a36-4d13-a7dc-6a42262136be": ["225534bb-e40d-42be-9258-309083656512"], "8f74dbe1-c3ed-48ca-9635-d701d26e829a": ["225534bb-e40d-42be-9258-309083656512"], "3809c393-b89e-494c-b529-c65e601c1544": ["52b00ce1-0f48-46fb-9bdb-6c3ab575940b"], "edb9b7b1-11c1-421c-a07f-7abe3d6e7c21": ["52b00ce1-0f48-46fb-9bdb-6c3ab575940b"], "5a48e740-85f0-48c7-b0c7-6247c384f052": ["3604ee55-dc85-43ef-8409-908fe897aef7"], "d6567db0-b18c-4dcb-b80c-146f2047bc13": ["3604ee55-dc85-43ef-8409-908fe897aef7"], "59a37c01-7bac-4f9d-980f-48f5489e61e6": ["760e42ec-824f-4c12-98b7-856008ae5680"], "e24a71f0-8b86-461a-92bd-fa6cef7ca33b": ["760e42ec-824f-4c12-98b7-856008ae5680"], "63dcc302-d64d-47f5-a304-a64d4d6642b4": ["706f37a3-1ae3-462f-9ae9-f447c8386d34"], "9b9b4805-12cb-453d-a3f4-ddbb20679c39": ["706f37a3-1ae3-462f-9ae9-f447c8386d34"], "7b3d457a-d0bf-4b13-b59c-df184af98f08": ["0d9098f6-5346-47fb-b91d-0a76054887ac"], "9473baea-32cd-4147-a547-5d45b0daa757": ["0d9098f6-5346-47fb-b91d-0a76054887ac"], "d4388801-831e-45e0-bf67-b67974027277": ["a5324dcc-7f7d-4d13-a7b4-c61a11b3471b"], "d4107956-2806-4098-a79e-e753cab1bf82": ["a5324dcc-7f7d-4d13-a7b4-c61a11b3471b"], "829774bb-4770-46cf-9f1b-86f51e7b6679": ["2c82cba7-cefa-41fd-a6d5-c90edb9b59f9"], "79c355b3-3945-402d-9d15-e460689ba635": ["2c82cba7-cefa-41fd-a6d5-c90edb9b59f9"], "e558dbd7-ca81-4070-9777-49636694d674": ["ab16f609-33d2-4f10-9b50-ff0066dc6a13"], "e1ce22f6-cad0-4bbe-87ae-5222158a4393": ["ab16f609-33d2-4f10-9b50-ff0066dc6a13"], "ae84398b-1649-4cce-8fa2-6295c80f7ec9": ["ff613344-c661-48a5-af0c-950d87f38882"], "648a7032-05c8-45c2-a7bb-2dca8fa9ffd0": ["ff613344-c661-48a5-af0c-950d87f38882"], "2b743770-5d66-4aa8-b9b4-c33adc78c1e3": ["ff7088b4-e4f7-4ef1-89b6-2293bc428ded"], "3b6f61ff-349d-4817-8c82-d064b9a71c86": ["ff7088b4-e4f7-4ef1-89b6-2293bc428ded"], "f596fded-c16b-49cb-b400-734c65b185af": ["c7bdee72-9ac2-418f-ac50-b41a38e31eb7"], "1626655d-7f72-4d0a-9170-3abdc8ed86ec": ["c7bdee72-9ac2-418f-ac50-b41a38e31eb7"], "cec6f35c-1b45-4d56-8c2f-aef7bc860a01": ["689778c9-90f6-4c4a-ab36-6fb05ad68144"], "12aca964-2112-4b36-8a40-14ab1512ac75": ["689778c9-90f6-4c4a-ab36-6fb05ad68144"], "53a48063-f4fb-482f-bd70-36915ec63956": ["2f4d5ac1-d6b0-48df-a313-39f40766a20c"], "7fdbbfed-73aa-45a8-9f1c-58ec2c0f3912": ["2f4d5ac1-d6b0-48df-a313-39f40766a20c"], "0ed0fb9c-47c4-4c7c-a5ae-d7e3a35670a1": ["473f218e-e471-4506-a9ba-a4840bcf9eb1"], "88297ffa-b5ca-460c-81ed-a61975ab39ef": ["473f218e-e471-4506-a9ba-a4840bcf9eb1"], "38409d77-4936-4266-a7f3-2d910d3bea91": ["3d2d1cf5-ddbb-40dc-a570-3f55f091e095"], "3d2d3a9e-a6a7-49f5-bdd8-5db95fc8b602": ["3d2d1cf5-ddbb-40dc-a570-3f55f091e095"], "ca685f83-ccd7-4a17-a31d-bfc648b58840": ["fcbeb8b3-4cff-4248-b03e-fc6879248660"], "ce1fdffd-851d-463e-8f24-4596865b62dc": ["fcbeb8b3-4cff-4248-b03e-fc6879248660"], "1a82989c-3ead-4aea-9098-53d3dca7f9b7": ["5ff1ba24-2f90-4f45-a3a3-6e1c50395575"], "a30ea710-3349-4357-8dcb-915f6c69f2da": ["5ff1ba24-2f90-4f45-a3a3-6e1c50395575"], "004b52ee-6a49-47d7-a4bd-77ec96fadc31": ["62a002de-0d3c-44dd-a41c-3fd464e4087a"], "a5ad1cc1-318a-4210-8838-22015d780344": ["62a002de-0d3c-44dd-a41c-3fd464e4087a"], "f05e4729-18f1-4664-9f41-2ad997f9d726": ["7a809df5-be14-43b9-9219-bb0b8d1f7d2c"], "81c90ac3-caf0-4c9d-8e02-8c62d26a047e": ["7a809df5-be14-43b9-9219-bb0b8d1f7d2c"], "0abf12fc-3e73-41e5-8594-5e2bb6ecdb24": ["1b4ea0b8-2883-4f20-8b10-198e6ad55155"], "e3abf868-922a-42e7-8c5a-b1ff0a353d39": ["1b4ea0b8-2883-4f20-8b10-198e6ad55155"], "55c79cd5-dee3-4e43-b8a3-839028518379": ["5d49e42f-479a-415f-8de0-91ebbd0e77df"], "456333eb-689e-4896-b2d4-0cf136672c77": ["5d49e42f-479a-415f-8de0-91ebbd0e77df"], "8834b86c-b1b9-43d6-92e0-3c64ca09e854": ["d8dc77d4-d7bc-40c8-bb38-e6f96f77391c"], "e84f1a90-e702-4594-84b8-5c5b67352195": ["d8dc77d4-d7bc-40c8-bb38-e6f96f77391c"], "490b6ca7-059f-41fe-82ae-b8d2c3890cf1": ["c3a79cf4-99fe-41a5-94a9-9972c547b027"], "59bed72b-bd80-47c3-bb57-08dd086ecf9d": ["c3a79cf4-99fe-41a5-94a9-9972c547b027"], "625e3e66-e1fc-4223-a201-e88b765f449e": ["ecf9714c-7e5b-4f00-9fad-45441a3db2a8"], "da4a10c9-db2a-45fa-bad5-b66ef842c023": ["ecf9714c-7e5b-4f00-9fad-45441a3db2a8"], "40ab1b55-bc53-4cae-8f7e-4657a5b2bdc2": ["e8c07b22-d96c-4cfc-be67-00e326b77e19"], "46de7819-7250-4050-8bf9-4635a1a02f3e": ["e8c07b22-d96c-4cfc-be67-00e326b77e19"], "6feae899-9900-454f-a64d-39e842af8c76": ["1787e4ab-ddaa-436b-a84c-5b09e0444b2b"], "36826afc-57e4-4d70-bc7e-4ca62e3e3e67": ["1787e4ab-ddaa-436b-a84c-5b09e0444b2b"], "84440495-e768-4885-b78b-d8a0c17f3809": ["963066ad-85cd-44d7-a513-c5fc3b5f1733"], "9c3a8107-d49c-4dc0-9f78-d71a506df892": ["963066ad-85cd-44d7-a513-c5fc3b5f1733"], "068d8bd2-9336-4e18-bd93-2199100e631f": ["5ad44c84-503d-4b61-95dc-22017c580f31"], "3138ca26-38b8-4e17-9b31-b38bc8a8eb4f": ["5ad44c84-503d-4b61-95dc-22017c580f31"], "095919bc-18fa-4316-b1e8-07572983b77b": ["ac5d591f-9174-44b6-be57-08f8b0e48100"], "39406f17-a757-4201-91b5-284ba4ebbd39": ["ac5d591f-9174-44b6-be57-08f8b0e48100"], "2744b9cf-981d-42e5-aed3-bb8e5acb0b2e": ["d41067f5-b199-46fa-95e6-571e133d23ff"], "798b53f4-f798-418a-abcd-6dd05f707c67": ["d41067f5-b199-46fa-95e6-571e133d23ff"], "d7fa2d65-26f8-4442-86f6-f1d6256e588a": ["c100cd93-2611-4d50-a99b-8728ccb99ba1"], "e50c31b3-bab1-4064-baa1-199c946d9789": ["c100cd93-2611-4d50-a99b-8728ccb99ba1"], "644dcaa5-1731-43fe-b0f5-c6a4bc05564e": ["0b2a13ab-790a-4e74-97a6-dbd3f2f3834d"], "def43eb9-80b0-4ad2-9198-d84ecb89c720": ["0b2a13ab-790a-4e74-97a6-dbd3f2f3834d"], "8495a23f-4bb7-47ac-8c54-58cf5675cdd7": ["c65eb4b9-10bb-4fcf-b682-fca84d3f37a1"], "74ae51e9-63b3-48ce-9be7-4f88052d7bd6": ["c65eb4b9-10bb-4fcf-b682-fca84d3f37a1"], "11cdd3ed-e09b-463d-9853-0be811073b75": ["2ac15af5-0f67-4ab6-803a-169153471fbe"], "9ca1ff0e-0cd9-4362-aca9-fd904077c845": ["2ac15af5-0f67-4ab6-803a-169153471fbe"], "8fe0054d-51ba-48c5-8cc5-259b2b96f535": ["c3a647af-08ee-42b7-87a6-57644e59b9eb"], "03b9f17b-0b61-401b-bc65-47d0655f31d8": ["c3a647af-08ee-42b7-87a6-57644e59b9eb"], "6d622041-fccf-4eb4-9a53-f7d7577856f8": ["9aa5eff7-f727-421e-835d-3def1111689a"], "a1738003-3e17-48e7-86a2-1410bc0f1c07": ["9aa5eff7-f727-421e-835d-3def1111689a"], "d15e0c10-378f-48a3-9a5c-be0c618106b4": ["ecb13fde-537f-49b6-82bd-ad0e6de18a8c"], "7e7e2c28-ea80-4568-a71a-41966f9f117f": ["ecb13fde-537f-49b6-82bd-ad0e6de18a8c"], "57073541-fc8c-43cd-8b42-f9497eb501af": ["8f297398-44b9-4be9-bbfb-ff90fef13d5f"], "92d9e36d-0fef-4b2e-b40d-ff2b800fcf10": ["8f297398-44b9-4be9-bbfb-ff90fef13d5f"], "6f7aa060-c19a-4614-83d2-134828a7e956": ["04e3f601-a4a2-4cc0-9978-8595281b3c94"], "6b95bc28-dbb4-408f-8c5b-f5b37073b6fd": ["04e3f601-a4a2-4cc0-9978-8595281b3c94"], "4776eaa1-b6f0-440c-a6be-923bbf49687d": ["6690225c-fbc4-4316-bef9-9cf1d5e5957c"], "acf74d86-1184-4092-8a1d-3ca58f5fe97a": ["6690225c-fbc4-4316-bef9-9cf1d5e5957c"], "2c1b02c6-1919-49ea-beff-165567d20b47": ["73043a09-91db-4768-9c0b-702c2dfcd9f0"], "2d15dfed-c66d-4fac-89dd-3aded02ec63e": ["73043a09-91db-4768-9c0b-702c2dfcd9f0"], "e71beb7c-7564-4f2c-83f7-ec9bb3134847": ["2cfdb40f-4c06-45c7-ab73-2bcc65986c58"], "6c079fa0-60c3-4c8d-826a-2816c65d3ea0": ["2cfdb40f-4c06-45c7-ab73-2bcc65986c58"], "543e9bfb-b5f4-4247-89c8-41e0e7fb11a9": ["65cc819a-a0c3-4ffa-b6f0-e47f846de5a5"], "8613b055-c817-4a59-84cf-1ae29a7c2269": ["65cc819a-a0c3-4ffa-b6f0-e47f846de5a5"], "ce252388-c4d9-4968-aadf-218b47f609a5": ["f258f74e-4463-4558-a8be-88fcc9da5b5a"], "f46088d7-1004-41cb-87c5-8a2b0bcdef59": ["f258f74e-4463-4558-a8be-88fcc9da5b5a"], "b5f49997-5049-4865-9b5b-c18d880e2baf": ["16d54bad-34c2-4427-a979-eb6a860bc22e"], "eeb5acfd-3be2-4488-b45e-e0979bd5c855": ["16d54bad-34c2-4427-a979-eb6a860bc22e"]}, "corpus": {"80e81c8c-bb97-4604-bdef-dcc56813587a": "-    \nUSING THIS TECHNICAL COMPANION\nThe Blueprint for an AI Bill of Rights is a set of five principles and associated practices to help guide the design, \nuse, and deployment of automated systems to protect the rights of the American public in the age of artificial \nintelligence. This technical companion considers each principle in the Blueprint for an AI Bill of Rights and \nprovides examples and concrete steps for communities, industry, governments, and others to take in order to \nbuild these protections into policy, practice, or the technological design process. \nTaken together, the technical protections and practices laid out in the Blueprint for an AI Bill of Rights can help \nguard the American public against many of the potential and actual harms identified by researchers, technolo\u00ad\ngists, advocates, journalists, policymakers, and communities in the United States and around the world. This", "d0a6097e-42c8-499f-8d6d-bcfae7f992d5": "via application programming interfaces). Independent evaluators, such as researchers, journalists, ethics \nreview boards, inspectors general, and third-party auditors, should be given access to the system and samples \nof associated data, in a manner consistent with privacy, security, law, or regulation (including, e.g., intellectual \nproperty law), in order to perform such evaluations. Mechanisms should be included to ensure that system \naccess for evaluation is: provided in a timely manner to the deployment-ready version of the system; trusted to \nprovide genuine, unfiltered access to the full system; and truly independent such that evaluator access cannot \nbe revoked without reasonable and verified justification. \nReporting.12 Entities responsible for the development or use of automated systems should provide \nregularly-updated reports that include: an overview of the system, including how it is embedded in the", "51421b31-1a41-49da-a2c2-65df54ae93ce": "requirement. \nProviding notice has long been a standard practice, and in many cases is a legal requirement, when, for example, \nmaking a video recording of someone (outside of a law enforcement or national security context). In some cases, such \nas credit, lenders are required to provide notice and explanation to consumers. Techniques used to automate the \nprocess of explaining such systems are under active research and improvement and such explanations can take many \nforms. Innovative companies and researchers are rising to the challenge and creating and deploying explanatory \nsystems that can help the public better understand decisions that impact them. \nWhile notice and explanation requirements are already in place in some sectors or situations, the American public \ndeserve to know consistently and across sectors if an automated system is being used in a way that impacts their rights, \nopportunities, or access. This knowledge should provide confidence in how the public is being treated, and trust in the", "758f783b-3fdc-4890-9de4-da3c035c1141": "than role models, toys, or activities.40 Some search engines have been working to reduce the prevalence of\nthese results, but the problem remains.41\n\u2022\nAdvertisement delivery systems that predict who is most likely to click on a job advertisement end up deliv-\nering ads in ways that reinforce racial and gender stereotypes, such as overwhelmingly directing supermar-\nket cashier ads to women and jobs with taxi companies to primarily Black people.42\u00ad\n\u2022\nBody scanners, used by TSA at airport checkpoints, require the operator to select a \u201cmale\u201d or \u201cfemale\u201d\nscanning setting based on the passenger\u2019s sex, but the setting is chosen based on the operator\u2019s perception of\nthe passenger\u2019s gender identity. These scanners are more likely to flag transgender travelers as requiring\nextra screening done by a person. Transgender travelers have described degrading experiences associated\nwith these extra screenings.43 TSA has recently announced plans to implement a gender-neutral algorithm44", "96838aa0-1bf7-4ae3-a8d7-5d093e9feb39": "ABOUT THIS FRAMEWORK\u00ad\u00ad\u00ad\u00ad\u00ad\nThe Blueprint for an AI Bill of Rights is a set of five principles and associated practices to help guide the \ndesign, use, and deployment of automated systems to protect the rights of the American public in the age of \nartificial intel-ligence. Developed through extensive consultation with the American public, these principles are \na blueprint for building and deploying automated systems that are aligned with democratic values and protect \ncivil rights, civil liberties, and privacy. The Blueprint for an AI Bill of Rights includes this Foreword, the five \nprinciples, notes on Applying the The Blueprint for an AI Bill of Rights, and a Technical Companion that gives \nconcrete steps that can be taken by many kinds of organizations\u2014from governments at all levels to companies of \nall sizes\u2014to uphold these values. Experts from across the private sector, governments, and international", "66c96cba-2674-4734-a869-d002faab751c": "in place, providing an important alternative to ensure access. Companies that have introduced automated call centers \noften retain the option of dialing zero to reach an operator. When automated identity controls are in place to board an \nairplane or enter the country, there is a person supervising the systems who can be turned to for help or to appeal a \nmisidentification. \nThe American people deserve the reassurance that such procedures are in place to protect their rights, opportunities, \nand access. People make mistakes, and a human alternative or fallback mechanism will not always have the right \nanswer, but they serve as an important check on the power and validity of automated systems. \n\u2022 An automated signature matching system is used as part of the voting process in many parts of the country to\ndetermine whether the signature on a mail-in ballot matches the signature on file. These signature matching\nsystems are less likely to work correctly for some voters, including voters with mental or physical", "2689bb50-4ffd-4610-856c-c8fad4ab7285": "data augmentations, parameter adjustments, or other modi\ufb01cations. Access to \nun-tuned (baseline) models supports debugging the relative in\ufb02uence of the pre-\ntrained weights compared to the \ufb01ne-tuned model weights or other system \nupdates. \nInformation Integrity; Data Privacy \nMG-3.2-003 \nDocument sources and types of training data and their origins, potential biases \npresent in the data related to the GAI application and its content provenance, \narchitecture, training process of the pre-trained model including information on \nhyperparameters, training duration, and any \ufb01ne-tuning or retrieval-augmented \ngeneration processes applied. \nInformation Integrity; Harmful Bias \nand Homogenization; Intellectual \nProperty \nMG-3.2-004 Evaluate user reported problematic content and integrate feedback into system \nupdates. \nHuman-AI Con\ufb01guration, \nDangerous, Violent, or Hateful \nContent \nMG-3.2-005 \nImplement content \ufb01lters to prevent the generation of inappropriate, harmful,", "7515dd00-b05d-49ea-baa0-7cedeb05eb39": "SECTION TITLE\nApplying The Blueprint for an AI Bill of Rights \nWhile many of the concerns addressed in this framework derive from the use of AI, the technical \ncapabilities and specific definitions of such systems change with the speed of innovation, and the potential \nharms of their use occur even with less technologically sophisticated tools. Thus, this framework uses a two-\npart test to determine what systems are in scope. This framework applies to (1) automated systems that (2) \nhave the potential to meaningfully impact the American public\u2019s rights, opportunities, or access to \ncritical resources or services. These rights, opportunities, and access to critical resources of services should \nbe enjoyed equally and be fully protected, regardless of the changing role that automated systems may play in \nour lives. \nThis framework describes protections that should be applied with respect to all automated systems that \nhave the potential to meaningfully impact individuals' or communities' exercise of:", "f339987a-b2cd-4258-85c5-a864712a9e98": "Content; Harmful Bias and \nHomogenization \nMP-5.1-005 Conduct adversarial role-playing exercises, GAI red-teaming, or chaos testing to \nidentify anomalous or unforeseen failure modes. \nInformation Security \nMP-5.1-006 \nPro\ufb01le threats and negative impacts arising from GAI systems interacting with, \nmanipulating, or generating content, and outlining known and potential \nvulnerabilities and the likelihood of their occurrence. \nInformation Security \nAI Actor Tasks: AI Deployment, AI Design, AI Development, AI Impact Assessment, A\ufb00ected Individuals and Communities, End-\nUsers, Operation and Monitoring", "673465c5-faf7-4ab1-86e0-d7cc5751143d": "61. See, e.g., Nir Kshetri. School surveillance of students via laptops may do more harm than good. The\nConversation. Jan. 21, 2022.\nhttps://theconversation.com/school-surveillance-of-students-via-laptops-may-do-more-harm-than\u00ad\ngood-170983; Matt Scherer. Warning: Bossware May be Hazardous to Your Health. Center for Democracy\n& Technology Report.\nhttps://cdt.org/wp-content/uploads/2021/07/2021-07-29-Warning-Bossware-May-Be-Hazardous-To\u00ad\nYour-Health-Final.pdf; Human Impact Partners and WWRC. The Public Health Crisis Hidden in Amazon\nWarehouses. HIP and WWRC report. Jan. 2021.\nhttps://humanimpact.org/wp-content/uploads/2021/01/The-Public-Health-Crisis-Hidden-In-Amazon\u00ad\nWarehouses-HIP-WWRC-01-21.pdf; Drew Harwell. Contract lawyers face a growing invasion of\nsurveillance programs that monitor their work. The Washington Post. Nov. 11, 2021. https://\nwww.washingtonpost.com/technology/2021/11/11/lawyer-facial-recognition-monitoring/;", "3df80c8e-fd5b-436c-9411-42e36faeeaef": "The Equal Employment Opportunity Commission and the Department of Justice have clearly \nlaid out how employers\u2019 use of AI and other automated systems can result in \ndiscrimination against job applicants and employees with disabilities.53 The documents explain \nhow employers\u2019 use of software that relies on algorithmic decision-making may violate existing requirements \nunder Title I of the Americans with Disabilities Act (\u201cADA\u201d). This technical assistance also provides practical \ntips to employers on how to comply with the ADA, and to job applicants and employees who think that their \nrights may have been violated. \nDisparity assessments identified harms to Black patients' healthcare access. A widely \nused healthcare algorithm relied on the cost of each patient\u2019s past medical care to predict future medical needs, \nrecommending early interventions for the patients deemed most at risk. This process discriminated", "225534bb-e40d-42be-9258-309083656512": "28 \nMAP 5.2: Practices and personnel for supporting regular engagement with relevant AI Actors and integrating feedback about \npositive, negative, and unanticipated impacts are in place and documented. \nAction ID \nSuggested Action \nGAI Risks \nMP-5.2-001 \nDetermine context-based measures to identify if new impacts are present due to \nthe GAI system, including regular engagements with downstream AI Actors to \nidentify and quantify new contexts of unanticipated impacts of GAI systems. \nHuman-AI Con\ufb01guration; Value \nChain and Component Integration \nMP-5.2-002 \nPlan regular engagements with AI Actors responsible for inputs to GAI systems, \nincluding third-party data and algorithms, to review and evaluate unanticipated \nimpacts. \nHuman-AI Con\ufb01guration; Value \nChain and Component Integration \nAI Actor Tasks: AI Deployment, AI Design, AI Impact Assessment, A\ufb00ected Individuals and Communities, Domain Experts, End-\nUsers, Human Factors, Operation and Monitoring", "52b00ce1-0f48-46fb-9bdb-6c3ab575940b": "and Homogenization \nGV-3.2-003 \nDe\ufb01ne acceptable use policies for GAI interfaces, modalities, and human-AI \ncon\ufb01gurations (i.e., for chatbots and decision-making tasks), including criteria for \nthe kinds of queries GAI applications should refuse to respond to.  \nHuman-AI Con\ufb01guration \nGV-3.2-004 \nEstablish policies for user feedback mechanisms for GAI systems which include \nthorough instructions and any mechanisms for recourse. \nHuman-AI Con\ufb01guration  \nGV-3.2-005 \nEngage in threat modeling to anticipate potential risks from GAI systems. \nCBRN Information or Capabilities; \nInformation Security \nAI Actors: AI Design \n \nGOVERN 4.1: Organizational policies and practices are in place to foster a critical thinking and safety-\ufb01rst mindset in the design, \ndevelopment, deployment, and uses of AI systems to minimize potential negative impacts. \nAction ID \nSuggested Action \nGAI Risks \nGV-4.1-001 \nEstablish policies and procedures that address continual improvement processes", "3604ee55-dc85-43ef-8409-908fe897aef7": "MEASURE 4.2: Measurement results regarding AI system trustworthiness in deployment context(s) and across the AI lifecycle are \ninformed by input from domain experts and relevant AI Actors to validate whether the system is performing consistently as \nintended. Results are documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-4.2-001 \nConduct adversarial testing at a regular cadence to map and measure GAI risks, \nincluding tests to address attempts to deceive or manipulate the application of \nprovenance techniques or other misuses. Identify vulnerabilities and \nunderstand potential misuse scenarios and unintended outputs. \nInformation Integrity; Information \nSecurity \nMS-4.2-002 \nEvaluate GAI system performance in real-world scenarios to observe its \nbehavior in practical environments and reveal issues that might not surface in \ncontrolled and optimized testing environments. \nHuman-AI Con\ufb01guration; \nConfabulation; Information \nSecurity \nMS-4.2-003", "760e42ec-824f-4c12-98b7-856008ae5680": "Demonstrate access to human alternatives, consideration, and fallback \nReporting. Reporting should include an assessment of timeliness and the extent of additional burden for \nhuman alternatives, aggregate statistics about who chooses the human alternative, along with the results of \nthe assessment about brevity, clarity, and accessibility of notice and opt-out instructions. Reporting on the \naccessibility, timeliness, and effectiveness of human consideration and fallback should be made public at regu\u00ad\nlar intervals for as long as the system is in use. This should include aggregated information about the number \nand type of requests for consideration, fallback employed, and any repeated requests; the timeliness of the \nhandling of these requests, including mean wait times for different types of requests as well as maximum wait \ntimes; and information about the procedures used to address requests for consideration along with the results", "706f37a3-1ae3-462f-9ae9-f447c8386d34": "protections should be built into their design, deployment, and ongoing use. \nMany companies, non-profits, and federal government agencies are already taking steps to ensure the public \nis protected from algorithmic discrimination. Some companies have instituted bias testing as part of their product \nquality assessment and launch procedures, and in some cases this testing has led products to be changed or not \nlaunched, preventing harm to the public. Federal government agencies have been developing standards and guidance \nfor the use of automated systems in order to help prevent bias. Non-profits and companies have developed best \npractices for audits and impact assessments to help identify potential algorithmic discrimination and provide \ntransparency to the public in the mitigation of such biases. \nBut there is much more work to do to protect the public from algorithmic discrimination to use and design", "0d9098f6-5346-47fb-b91d-0a76054887ac": "voting, and protections from discrimination, excessive punishment, unlawful surveillance, and violations of \nprivacy and other freedoms in both public and private sector contexts; equal opportunities, including equitable \naccess to education, housing, credit, employment, and other programs; or, access to critical resources or \nservices, such as healthcare, financial services, safety, social services, non-deceptive information about goods \nand services, and government benefits. \n10", "a5324dcc-7f7d-4d13-a7b4-c61a11b3471b": "FROM \nPRINCIPLES \nTO PRACTICE \nA TECHINCAL COMPANION TO\nTHE Blueprint for an \nAI BILL OF RIGHTS\n12", "2c82cba7-cefa-41fd-a6d5-c90edb9b59f9": "reuse \nRelevant and high-quality data. Data used as part of any automated system\u2019s creation, evaluation, or \ndeployment should be relevant, of high quality, and tailored to the task at hand. Relevancy should be \nestablished based on research-backed demonstration of the causal influence of the data to the specific use case \nor justified more generally based on a reasonable expectation of usefulness in the domain and/or for the \nsystem design or ongoing development. Relevance of data should not be established solely by appealing to \nits historical connection to the outcome. High quality and tailored data should be representative of the task at \nhand and errors from data entry or other sources should be measured and limited. Any data used as the target \nof a prediction process should receive particular attention to the quality and validity of the predicted outcome \nor label to ensure the goal of the automated system is appropriately identified and measured. Additionally,", "ab16f609-33d2-4f10-9b50-ff0066dc6a13": "measured quantitatively, including explanations as to why some risks cannot be \nmeasured (e.g., due to technological limitations, resource constraints, or \ntrustworthy considerations). Include unmeasured risks in marginal risks. \nInformation Integrity \nAI Actor Tasks: AI Development, Domain Experts, TEVV \n \nMEASURE 1.3: Internal experts who did not serve as front-line developers for the system and/or independent assessors are \ninvolved in regular assessments and updates. Domain experts, users, AI Actors external to the team that developed or deployed the \nAI system, and a\ufb00ected communities are consulted in support of assessments as necessary per organizational risk tolerance. \nAction ID \nSuggested Action \nGAI Risks \nMS-1.3-001 \nDe\ufb01ne relevant groups of interest (e.g., demographic groups, subject matter \nexperts, experience with GAI technology) within the context of use as part of \nplans for gathering structured public feedback. \nHuman-AI Con\ufb01guration; Harmful \nBias and Homogenization; CBRN", "ff613344-c661-48a5-af0c-950d87f38882": "it comes to open-ended prompts for long-form responses and in domains which require highly \ncontextual and/or domain expertise.  \nRisks from confabulations may arise when users believe false content \u2013 often due to the con\ufb01dent nature \nof the response \u2013 leading users to act upon or promote the false information. This poses a challenge for \nmany real-world applications, such as in healthcare, where a confabulated summary of patient \ninformation reports could cause doctors to make incorrect diagnoses and/or recommend the wrong \ntreatments. Risks of confabulated content may be especially important to monitor when integrating GAI \ninto applications involving consequential decision making. \nGAI outputs may also include confabulated logic or citations that purport to justify or explain the \nsystem\u2019s answer, which may further mislead humans into inappropriately trusting the system\u2019s output. \nFor instance, LLMs sometimes provide logical steps for how they arrived at an answer even when the", "ff7088b4-e4f7-4ef1-89b6-2293bc428ded": "resulting data to surveil individual employees and surreptitiously intervene in discussions.67\n32", "c7bdee72-9ac2-418f-ac50-b41a38e31eb7": "\u2022\nPamela Wisniewski, Associate Professor of Computer Science, University of Central Florida; Director,\nSocio-technical Interaction Research (STIR) Lab\n\u2022\nSeny Kamara, Associate Professor of Computer Science, Brown University\nEach panelist individually emphasized the risks of using AI in high-stakes settings, including the potential for \nbiased data and discriminatory outcomes, opaque decision-making processes, and lack of public trust and \nunderstanding of the algorithmic systems. The interventions and key needs various panelists put forward as \nnecessary to the future design of critical AI systems included ongoing transparency, value sensitive and \nparticipatory design, explanations designed for relevant stakeholders, and public consultation. \nVarious \npanelists emphasized the importance of placing trust in people, not technologies, and in engaging with \nimpacted communities to understand the potential harms of technologies and build protection by design into \nfuture systems.", "689778c9-90f6-4c4a-ab36-6fb05ad68144": "The demographics of the assessed groups should be as inclusive as possible of race, color, ethnicity, sex \n(including pregnancy, childbirth, and related medical conditions, gender identity, intersex status, and sexual \norientation), religion, age, national origin, disability, veteran status, genetic information, or any other classifi\u00ad\ncation protected by law. The broad set of measures assessed should include demographic performance mea\u00ad\nsures, overall and subgroup parity assessment, and calibration. Demographic data collected for disparity \nassessment should be separated from data used for the automated system and privacy protections should be \ninstituted; in some cases it may make sense to perform such assessment using a data sample. For every \ninstance where the deployed automated system leads to different treatment or impacts disfavoring the identi\u00ad\nfied groups, the entity governing, implementing, or using the system should document the disparity and a \njustification for any continued use of the system.", "2f4d5ac1-d6b0-48df-a313-39f40766a20c": "future systems. \nPanel 5: Social Welfare and Development. This event explored current and emerging uses of technology to \nimplement or improve social welfare systems, social development programs, and other systems that can impact \nlife chances. \nWelcome:\n\u2022\nSuresh Venkatasubramanian, Assistant Director for Science and Justice, White House Office of Science\nand Technology Policy\n\u2022\nAnne-Marie Slaughter, CEO, New America\nModerator: Michele Evermore, Deputy Director for Policy, Office of Unemployment Insurance \nModernization, Office of the Secretary, Department of Labor \nPanelists:\n\u2022\nBlake Hall, CEO and Founder, ID.Me\n\u2022\nKarrie Karahalios, Professor of Computer Science, University of Illinois, Urbana-Champaign\n\u2022\nChristiaan van Veen, Director of Digital Welfare State and Human Rights Project, NYU School of Law's\nCenter for Human Rights and Global Justice\n58", "473f218e-e471-4506-a9ba-a4840bcf9eb1": "and data agency can be meaningful and not overwhelming. These choices\u2014such as contextual, timely \nalerts about location tracking\u2014are brief, direct, and use-specific. Many of the expectations listed here for \nprivacy by design and use-specific consent mirror those distributed to developers as best practices when \ndeveloping for smart phone devices,82 such as being transparent about how user data will be used, asking for app \npermissions during their use so that the use-context will be clear to users, and ensuring that the app will still \nwork if users deny (or later revoke) some permissions. \n39", "3d2d1cf5-ddbb-40dc-a570-3f55f091e095": "the privacy, civil rights, and civil liberties implications of the use of such technologies be issued before \nbiometric identification technologies can be used in New York schools. \nFederal law requires employers, and any consultants they may retain, to report the costs \nof surveilling employees in the context of a labor dispute, providing a transparency \nmechanism to help protect worker organizing. Employers engaging in workplace surveillance \"where \nan object there-of, directly or indirectly, is [\u2026] to obtain information concerning the activities of employees or a \nlabor organization in connection with a labor dispute\" must report expenditures relating to this surveillance to \nthe Department of Labor Office of Labor-Management Standards, and consultants who employers retain for \nthese purposes must also file reports regarding their activities.81\nPrivacy choices on smartphones show that when technologies are well designed, privacy", "fcbeb8b3-4cff-4248-b03e-fc6879248660": "AI BILL OF RIGHTS\nFFECTIVE SYSTEMS\nineffective systems. Automated systems should be \ncommunities, stakeholders, and domain experts to identify \nSystems should undergo pre-deployment testing, risk \nthat demonstrate they are safe and effective based on \nincluding those beyond the intended use, and adherence to \nprotective measures should include the possibility of not \nAutomated systems should not be designed with an intent \nreasonably foreseeable possibility of endangering your safety or the safety of your community. They should \nstemming from unintended, yet foreseeable, uses or \n \n \n \n \n  \n \n \nSECTION TITLE\nBLUEPRINT FOR AN\nSAFE AND E \nYou should be protected from unsafe or \ndeveloped with consultation from diverse \nconcerns, risks, and potential impacts of the system. \nidentification and mitigation, and ongoing monitoring \ntheir intended use, mitigation of unsafe outcomes \ndomain-specific standards. Outcomes of these \ndeploying the system or removing a system from use. \nor", "5ff1ba24-2f90-4f45-a3a3-6e1c50395575": "which leads to extensive reuse of limited numbers of models; and the extent to which GAI may be \nintegrated into other devices and services. As GAI systems often involve many distinct third-party \ncomponents and data sources, it may be di\ufb03cult to attribute issues in a system\u2019s behavior to any one of \nthese sources. \nErrors in third-party GAI components can also have downstream impacts on accuracy and robustness. \nFor example, test datasets commonly used to benchmark or validate models can contain label errors. \nInaccuracies in these labels can impact the \u201cstability\u201d or robustness of these benchmarks, which many \nGAI practitioners consider during the model selection process.  \nTrustworthy AI Characteristics: Accountable and Transparent, Explainable and Interpretable, Fair with \nHarmful Bias Managed, Privacy Enhanced, Safe, Secure and Resilient, Valid and Reliable \n3. \nSuggested Actions to Manage GAI Risks \nThe following suggested actions target risks unique to or exacerbated by GAI.", "62a002de-0d3c-44dd-a41c-3fd464e4087a": "Confabulation; Harmful Bias and \nHomogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Governance and Oversight, Operation and Monitoring \n \nMANAGE 2.3: Procedures are followed to respond to and recover from a previously unknown risk when it is identi\ufb01ed. \nAction ID \nSuggested Action \nGAI Risks \nMG-2.3-001 \nDevelop and update GAI system incident response and recovery plans and \nprocedures to address the following: Review and maintenance of policies and \nprocedures to account for newly encountered uses; Review and maintenance of \npolicies and procedures for detection of unanticipated uses; Verify response \nand recovery plans account for the GAI system value chain; Verify response and \nrecovery plans are updated for and include necessary details to communicate \nwith downstream GAI system Actors: Points-of-Contact (POC), Contact \ninformation, noti\ufb01cation format. \nValue Chain and Component \nIntegration \nAI Actor Tasks: AI Deployment, Operation and Monitoring", "7a809df5-be14-43b9-9219-bb0b8d1f7d2c": "37 \nMS-2.11-005 \nAssess the proportion of synthetic to non-synthetic training data and verify \ntraining data is not overly homogenous or GAI-produced to mitigate concerns of \nmodel collapse. \nHarmful Bias and Homogenization \nAI Actor Tasks: AI Deployment, AI Impact Assessment, A\ufb00ected Individuals and Communities, Domain Experts, End-Users, \nOperation and Monitoring, TEVV \n \nMEASURE 2.12: Environmental impact and sustainability of AI model training and management activities \u2013 as identi\ufb01ed in the MAP \nfunction \u2013 are assessed and documented. \nAction ID \nSuggested Action \nGAI Risks \nMS-2.12-001 Assess safety to physical environments when deploying GAI systems. \nDangerous, Violent, or Hateful \nContent \nMS-2.12-002 Document anticipated environmental impacts of model development, \nmaintenance, and deployment in product design decisions. \nEnvironmental \nMS-2.12-003 \nMeasure or estimate environmental impacts (e.g., energy and water \nconsumption) for training, \ufb01ne tuning, and deploying models: Verify tradeo\ufb00s", "1b4ea0b8-2883-4f20-8b10-198e6ad55155": "47 \nAppendix A. Primary GAI Considerations \nThe following primary considerations were derived as overarching themes from the GAI PWG \nconsultation process. These considerations (Governance, Pre-Deployment Testing, Content Provenance, \nand Incident Disclosure) are relevant for voluntary use by any organization designing, developing, and \nusing GAI and also inform the Actions to Manage GAI risks. Information included about the primary \nconsiderations is not exhaustive, but highlights the most relevant topics derived from the GAI PWG.  \nAcknowledgments: These considerations could not have been surfaced without the helpful analysis and \ncontributions from the community and NIST sta\ufb00 GAI PWG leads: George Awad, Luca Belli, Harold Booth, \nMat Heyman, Yooyoung Lee, Mark Pryzbocki, Reva Schwartz, Martin Stanley, and Kyra Yee. \nA.1. Governance \nA.1.1. Overview \nLike any other technology system, governance principles and techniques can be used to manage risks", "5d49e42f-479a-415f-8de0-91ebbd0e77df": "Information Security; Value Chain \nand Component Integration \nAI Actor Tasks: AI Deployment, Operation and Monitoring, TEVV, Third-party entities \n \nMAP 1.1: Intended purposes, potentially bene\ufb01cial uses, context speci\ufb01c laws, norms and expectations, and prospective settings in \nwhich the AI system will be deployed are understood and documented. Considerations include: the speci\ufb01c set or types of users \nalong with their expectations; potential positive and negative impacts of system uses to individuals, communities, organizations, \nsociety, and the planet; assumptions and related limitations about AI system purposes, uses, and risks across the development or \nproduct AI lifecycle; and related TEVV and system metrics. \nAction ID \nSuggested Action \nGAI Risks \nMP-1.1-001 \nWhen identifying intended purposes, consider factors such as internal vs. \nexternal use, narrow vs. broad application scope, \ufb01ne-tuning, and varieties of \ndata sources (e.g., grounding, retrieval-augmented generation).", "d8dc77d4-d7bc-40c8-bb38-e6f96f77391c": "41 \nMG-2.2-006 \nUse feedback from internal and external AI Actors, users, individuals, and \ncommunities, to assess impact of AI-generated content. \nHuman-AI Con\ufb01guration \nMG-2.2-007 \nUse real-time auditing tools where they can be demonstrated to aid in the \ntracking and validation of the lineage and authenticity of AI-generated data. \nInformation Integrity \nMG-2.2-008 \nUse structured feedback mechanisms to solicit and capture user input about AI-\ngenerated content to detect subtle shifts in quality or alignment with \ncommunity and societal values. \nHuman-AI Con\ufb01guration; Harmful \nBias and Homogenization \nMG-2.2-009 \nConsider opportunities to responsibly use synthetic data and other privacy \nenhancing techniques in GAI development, where appropriate and applicable, \nmatch the statistical properties of real-world data without disclosing personally \nidenti\ufb01able information or contributing to homogenization. \nData Privacy; Intellectual Property; \nInformation Integrity; \nConfabulation; Harmful Bias and", "c3a79cf4-99fe-41a5-94a9-9972c547b027": "https://arxiv.org/pdf/2202.07646 \nCarlini, N. et al. (2024) Stealing Part of a Production Language Model. arXiv. \nhttps://arxiv.org/abs/2403.06634 \nChandra, B. et al. (2023) Dismantling the Disinformation Business of Chinese In\ufb02uence Operations. \nRAND. https://www.rand.org/pubs/commentary/2023/10/dismantling-the-disinformation-business-of-\nchinese.html \nCiriello, R. et al. (2024) Ethical Tensions in Human-AI Companionship: A Dialectical Inquiry into Replika. \nResearchGate. https://www.researchgate.net/publication/374505266_Ethical_Tensions_in_Human-\nAI_Companionship_A_Dialectical_Inquiry_into_Replika \nDahl, M. et al. (2024) Large Legal Fictions: Pro\ufb01ling Legal Hallucinations in Large Language Models. arXiv. \nhttps://arxiv.org/abs/2401.01301", "ecf9714c-7e5b-4f00-9fad-45441a3db2a8": "Electronic Privacy Information \nCenter (EPIC) \nEncode Justice \nEqual AI \nGoogle \nHitachi's AI Policy Committee \nThe Innocence Project \nInstitute of Electrical and \nElectronics Engineers (IEEE) \nIntuit \nLawyers Committee for Civil Rights \nUnder Law \nLegal Aid Society \nThe Leadership Conference on \nCivil and Human Rights \nMeta \nMicrosoft \nThe MIT AI Policy Forum \nMovement Alliance Project \nThe National Association of \nCriminal Defense Lawyers \nO\u2019Neil Risk Consulting & \nAlgorithmic Auditing \nThe Partnership on AI \nPinterest \nThe Plaintext Group \npymetrics \nSAP \nThe Security Industry Association \nSoftware and Information Industry \nAssociation (SIIA) \nSpecial Competitive Studies Project \nThorn \nUnited for Respect \nUniversity of California at Berkeley \nCitris Policy Lab \nUniversity of California at Berkeley \nLabor Center \nUnfinished/Project Liberty \nUpturn \nUS Chamber of Commerce \nUS Chamber of Commerce \nTechnology Engagement Center \nA.I. Working Group\nVibrent Health\nWarehouse Worker Resource\nCenter\nWaymap\n62", "e8c07b22-d96c-4cfc-be67-00e326b77e19": "APPENDIX\nLisa Feldman Barrett \nMadeline Owens \nMarsha Tudor \nMicrosoft Corporation \nMITRE Corporation \nNational Association for the \nAdvancement of Colored People \nLegal Defense and Educational \nFund \nNational Association of Criminal \nDefense Lawyers \nNational Center for Missing & \nExploited Children \nNational Fair Housing Alliance \nNational Immigration Law Center \nNEC Corporation of America \nNew America\u2019s Open Technology \nInstitute \nNew York Civil Liberties Union \nNo Name Provided \nNotre Dame Technology Ethics \nCenter \nOffice of the Ohio Public Defender \nOnfido \nOosto \nOrissa Rose \nPalantir \nPangiam \nParity Technologies \nPatrick A. Stewart, Jeffrey K. \nMullins, and Thomas J. Greitens \nPel Abbott \nPhiladelphia Unemployment \nProject \nProject On Government Oversight \nRecording Industry Association of \nAmerica \nRobert Wilkens \nRon Hedges \nScience, Technology, and Public \nPolicy Program at University of \nMichigan Ann Arbor \nSecurity Industry Association \nSheila Dean \nSoftware & Information Industry \nAssociation", "1787e4ab-ddaa-436b-a84c-5b09e0444b2b": "likelihood of such an attack. The physical synthesis development, production, and use of chemical or \nbiological agents will continue to require both applicable expertise and supporting materials and \ninfrastructure. The impact of GAI on chemical or biological agent misuse will depend on what the key \nbarriers for malicious actors are (e.g., whether information access is one such barrier), and how well GAI \ncan help actors address those barriers.  \nFurthermore, chemical and biological design tools (BDTs) \u2013 highly specialized AI systems trained on \nscienti\ufb01c data that aid in chemical and biological design \u2013 may augment design capabilities in chemistry \nand biology beyond what text-based LLMs are able to provide. As these models become more \ne\ufb03cacious, including for bene\ufb01cial uses, it will be important to assess their potential to be used for \nharm, such as the ideation and design of novel harmful chemical or biological agents.", "963066ad-85cd-44d7-a513-c5fc3b5f1733": "51 \ngeneral public participants. For example, expert AI red-teamers could modify or verify the \nprompts written by general public AI red-teamers. These approaches may also expand coverage \nof the AI risk attack surface.  \n\u2022 \nHuman / AI: Performed by GAI in combination with specialist or non-specialist human teams. \nGAI-led red-teaming can be more cost e\ufb00ective than human red-teamers alone. Human or GAI-\nled AI red-teaming may be better suited for eliciting di\ufb00erent types of harms. \n \nA.1.6. Content Provenance \nOverview \nGAI technologies can be leveraged for many applications such as content generation and synthetic data. \nSome aspects of GAI outputs, such as the production of deepfake content, can challenge our ability to \ndistinguish human-generated content from AI-generated synthetic content. To help manage and mitigate \nthese risks, digital transparency mechanisms like provenance data tracking can trace the origin and", "5ad44c84-503d-4b61-95dc-22017c580f31": "to a particular decision, and should be meaningful for the particular customization based on purpose, target, \nand level of risk. While approximation and simplification may be necessary for the system to succeed based on \nthe explanatory purpose and target of the explanation, or to account for the risk of fraud or other concerns \nrelated to revealing decision-making information, such simplifications should be done in a scientifically \nsupportable way. Where appropriate based on the explanatory system, error ranges for the explanation should \nbe calculated and included in the explanation, with the choice of presentation of such information balanced \nwith usability and overall interface complexity concerns. \nDemonstrate protections for notice and explanation \nReporting. Summary reporting should document the determinations made based on the above consider\u00ad\nations, including: the responsible entities for accountability purposes; the goal and use cases for the system,", "ac5d591f-9174-44b6-be57-08f8b0e48100": "and Technology Policy\n\u2022\nBen Winters, Counsel, Electronic Privacy Information Center\nModerator: Chiraag Bains, Deputy Assistant to the President on Racial Justice & Equity \nPanelists: \n\u2022\nSean Malinowski, Director of Policing Innovation and Reform, University of Chicago Crime Lab\n\u2022\nKristian Lum, Researcher\n\u2022\nJumana Musa, Director, Fourth Amendment Center, National Association of Criminal Defense Lawyers\n\u2022\nStanley Andrisse, Executive Director, From Prison Cells to PHD; Assistant Professor, Howard University\nCollege of Medicine\n\u2022\nMyaisha Hayes, Campaign Strategies Director, MediaJustice\nPanelists discussed uses of technology within the criminal justice system, including the use of predictive \npolicing, pretrial risk assessments, automated license plate readers, and prison communication tools. The \ndiscussion emphasized that communities deserve safety, and strategies need to be identified that lead to safety; \nsuch strategies might include data-driven approaches, but the focus on safety should be primary, and", "d41067f5-b199-46fa-95e6-571e133d23ff": "ENDNOTES\n12. Expectations about reporting are intended for the entity developing or using the automated system. The\nresulting reports can be provided to the public, regulators, auditors, industry standards groups, or others\nengaged in independent review, and should be made public as much as possible consistent with law,\nregulation, and policy, and noting that intellectual property or law enforcement considerations may prevent\npublic release. These reporting expectations are important for transparency, so the American people can\nhave confidence that their rights, opportunities, and access as well as their expectations around\ntechnologies are respected.\n13. National Artificial Intelligence Initiative Office. Agency Inventories of AI Use Cases. Accessed Sept. 8,\n2022. https://www.ai.gov/ai-use-case-inventories/\n14. National Highway Traffic Safety Administration. https://www.nhtsa.gov/\n15. See, e.g., Charles Pruitt. People Doing What They Do Best: The Professional Engineers and NHTSA. Public", "c100cd93-2611-4d50-a99b-8728ccb99ba1": "qualitative user experience research. Riskier and higher-impact systems should be monitored and assessed \nmore frequently. Outcomes of this assessment should include additional disparity mitigation, if needed, or \nfallback to earlier procedures in the case that equity standards are no longer met and can't be mitigated, and \nprior mechanisms provide better adherence to equity standards. \n27\nAlgorithmic \nDiscrimination \nProtections", "0b2a13ab-790a-4e74-97a6-dbd3f2f3834d": "Action ID \nSuggested Action \nGAI Risks \nGV-1.3-001 \nConsider the following factors when updating or de\ufb01ning risk tiers for GAI: Abuses \nand impacts to information integrity; Dependencies between GAI and other IT or \ndata systems; Harm to fundamental rights or public safety; Presentation of \nobscene, objectionable, o\ufb00ensive, discriminatory, invalid or untruthful output; \nPsychological impacts to humans (e.g., anthropomorphization, algorithmic \naversion, emotional entanglement); Possibility for malicious use; Whether the \nsystem introduces signi\ufb01cant new security vulnerabilities; Anticipated system \nimpact on some groups compared to others; Unreliable decision making \ncapabilities, validity, adaptability, and variability of GAI system performance over \ntime. \nInformation Integrity; Obscene, \nDegrading, and/or Abusive \nContent; Value Chain and \nComponent Integration; Harmful \nBias and Homogenization; \nDangerous, Violent, or Hateful \nContent; CBRN Information or \nCapabilities \nGV-1.3-002", "c65eb4b9-10bb-4fcf-b682-fca84d3f37a1": "Harmful Bias and Homogenization \nGV-6.2-006 \nEstablish policies and procedures to test and manage risks related to rollover and \nfallback technologies for GAI systems, acknowledging that rollover and fallback \nmay include manual processing. \nInformation Integrity \nGV-6.2-007 \nReview vendor contracts and avoid arbitrary or capricious termination of critical \nGAI technologies or vendor services and non-standard terms that may amplify or \ndefer liability in unexpected ways and/or contribute to unauthorized data \ncollection by vendors or third-parties (e.g., secondary data use). Consider: Clear \nassignment of liability and responsibility for incidents, GAI system changes over \ntime (e.g., \ufb01ne-tuning, drift, decay); Request: Noti\ufb01cation and disclosure for \nserious incidents arising from third-party data and systems; Service Level \nAgreements (SLAs) in vendor contracts that address incident response, response \ntimes, and availability of critical support. \nHuman-AI Con\ufb01guration; \nInformation Security; Value Chain", "2ac15af5-0f67-4ab6-803a-169153471fbe": "and the integrity and (when applicable) the con\ufb01dentiality of the GAI code, training data, and model \nweights. To identify and secure potential attack points in AI systems or speci\ufb01c components of the AI \n \n \n12 See also https://doi.org/10.6028/NIST.AI.100-4, to be published.", "c3a647af-08ee-42b7-87a6-57644e59b9eb": "\u2022\nA company installed AI-powered cameras in its delivery vans in order to evaluate the road safety habits of its driv\u00ad\ners, but the system incorrectly penalized drivers when other cars cut them off or when other events beyond\ntheir control took place on the road. As a result, drivers were incorrectly ineligible to receive a bonus.11\n17", "9aa5eff7-f727-421e-835d-3def1111689a": "between resources used at inference time versus additional resources required \nat training time. \nEnvironmental \nMS-2.12-004 Verify e\ufb00ectiveness of carbon capture or o\ufb00set programs for GAI training and \napplications, and address green-washing concerns. \nEnvironmental \nAI Actor Tasks: AI Deployment, AI Impact Assessment, Domain Experts, Operation and Monitoring, TEVV", "ecb13fde-537f-49b6-82bd-ad0e6de18a8c": "17 \nGOVERN 1.7: Processes and procedures are in place for decommissioning and phasing out AI systems safely and in a manner that \ndoes not increase risks or decrease the organization\u2019s trustworthiness. \nAction ID \nSuggested Action \nGAI Risks \nGV-1.7-001 Protocols are put in place to ensure GAI systems are able to be deactivated when \nnecessary.  \nInformation Security; Value Chain \nand Component Integration \nGV-1.7-002 \nConsider the following factors when decommissioning GAI systems: Data \nretention requirements; Data security, e.g., containment, protocols, Data leakage \nafter decommissioning; Dependencies between upstream, downstream, or other \ndata, internet of things (IOT) or AI systems; Use of open-source data or models; \nUsers\u2019 emotional entanglement with GAI functions. \nHuman-AI Con\ufb01guration; \nInformation Security; Value Chain \nand Component Integration \nAI Actor Tasks: AI Deployment, Operation and Monitoring", "8f297398-44b9-4be9-bbfb-ff90fef13d5f": "shared, or made public as part of data brokerage or other agreements. Sensitive data includes data that can be \nused to infer sensitive information; even systems that are not directly marketed as sensitive domain technologies \nare expected to keep sensitive data private. Access to such data should be limited based on necessity and based \non a principle of local control, such that those individuals closest to the data subject have more access while \nthose who are less proximate do not (e.g., a teacher has access to their students\u2019 daily progress data while a \nsuperintendent does not). \nReporting. In addition to the reporting on data privacy (as listed above for non-sensitive data), entities devel-\noping technologies related to a sensitive domain and those collecting, using, storing, or sharing sensitive data \nshould, whenever appropriate, regularly provide public reports describing: any data security lapses or breaches", "04e3f601-a4a2-4cc0-9978-8595281b3c94": "APPENDIX\nSummaries of Additional Engagements: \n\u2022 OSTP created an email address ([email protected]) to solicit comments from the public on the use of\nartificial intelligence and other data-driven technologies in their lives.\n\u2022 OSTP issued a Request For Information (RFI) on the use and governance of biometric technologies.113 The\npurpose of this RFI was to understand the extent and variety of biometric technologies in past, current, or\nplanned use; the domains in which these technologies are being used; the entities making use of them; current\nprinciples, practices, or policies governing their use; and the stakeholders that are, or may be, impacted by their\nuse or regulation. The 130 responses to this RFI are available in full online114 and were submitted by the below\nlisted organizations and individuals:\nAccenture \nAccess Now \nACT | The App Association \nAHIP \nAIethicist.org \nAirlines for America \nAlliance for Automotive Innovation \nAmelia Winger-Bearskin \nAmerican Civil Liberties Union", "6690225c-fbc4-4316-bef9-9cf1d5e5957c": "ethics, or risk management. The Technical Companion builds on this prior work to provide practical next \nsteps to move these principles into practice and promote common approaches that allow technological \ninnovation to flourish while protecting people from harm. \n9", "73043a09-91db-4768-9c0b-702c2dfcd9f0": "Automated system support. Entities designing, developing, and deploying automated systems should \nestablish and maintain the capabilities that will allow individuals to use their own automated systems to help \nthem make consent, access, and control decisions in a complex data ecosystem. Capabilities include machine \nreadable data, standardized data formats, metadata or tags for expressing data processing permissions and \npreferences and data provenance and lineage, context of use and access-specific tags, and training models for \nassessing privacy risk. \nDemonstrate that data privacy and user control are protected \nIndependent evaluation. As described in the section on Safe and Effective Systems, entities should allow \nindependent evaluation of the claims made regarding data policies. These independent evaluations should be \nmade public whenever possible. Care will need to be taken to balance individual privacy with evaluation data \naccess needs.", "2cfdb40f-4c06-45c7-ab73-2bcc65986c58": "Information Integrity \nMS-2.7-006 \nMeasure the rate at which recommendations from security checks and incidents \nare implemented. Assess how quickly the AI system can adapt and improve \nbased on lessons learned from security incidents and feedback. \nInformation Integrity; Information \nSecurity \nMS-2.7-007 \nPerform AI red-teaming to assess resilience against: Abuse to facilitate attacks on \nother systems (e.g., malicious code generation, enhanced phishing content), GAI \nattacks (e.g., prompt injection), ML attacks (e.g., adversarial examples/prompts, \ndata poisoning, membership inference, model extraction, sponge examples). \nInformation Security; Harmful Bias \nand Homogenization; Dangerous, \nViolent, or Hateful Content \nMS-2.7-008 Verify \ufb01ne-tuning does not compromise safety and security controls. \nInformation Integrity; Information \nSecurity; Dangerous, Violent, or \nHateful Content", "65cc819a-a0c3-4ffa-b6f0-e47f846de5a5": "46 \nMG-4.3-003 \nReport GAI incidents in compliance with legal and regulatory requirements (e.g., \nHIPAA breach reporting, e.g., OCR (2023) or NHTSA (2022) autonomous vehicle \ncrash reporting requirements. \nInformation Security; Data Privacy \nAI Actor Tasks: AI Deployment, A\ufb00ected Individuals and Communities, Domain Experts, End-Users, Human Factors, Operation and \nMonitoring", "f258f74e-4463-4558-a8be-88fcc9da5b5a": "justification should be documented for each data attribute and source to explain why it is appropriate to use \nthat data to inform the results of the automated system and why such use will not violate any applicable laws. \nIn cases of high-dimensional and/or derived attributes, such justifications can be provided as overall \ndescriptions of the attribute generation process and appropriateness. \n19", "16d54bad-34c2-4427-a979-eb6a860bc22e": "related to generative AI models, capabilities, and applications. Organizations may choose to apply their \nexisting risk tiering to GAI systems, or they may opt to revise or update AI system risk levels to address \nthese unique GAI risks. This section describes how organizational governance regimes may be re-\nevaluated and adjusted for GAI contexts. It also addresses third-party considerations for governing across \nthe AI value chain.  \nA.1.2. Organizational Governance \nGAI opportunities, risks and long-term performance characteristics are typically less well-understood \nthan non-generative AI tools and may be perceived and acted upon by humans in ways that vary greatly. \nAccordingly, GAI may call for di\ufb00erent levels of oversight from AI Actors or di\ufb00erent human-AI \ncon\ufb01gurations in order to manage their risks e\ufb00ectively. Organizations\u2019 use of GAI systems may also \nwarrant additional human review, tracking and documentation, and greater management oversight."}}

Tasks/Task 4/val_questions.json

@@ -0,0 +1 @@
+{"61c5d3ec-11af-4a36-a028-e9e22afb5a8f": "What are the five principles outlined in the Blueprint for an AI Bill of Rights?", "662911a9-7407-4a24-95ff-2350dde354be": "How can communities and industry implement the practices suggested in the Blueprint for an AI Bill of Rights?", "09ded63e-b364-42e7-9677-e1dfa4932b9b": "What are the best practices for providing independent evaluators access to automated systems while ensuring privacy and security?", "6b539180-33d5-4cd2-abf5-63cbe6178e6a": "How can organizations ensure that evaluator access to automated systems remains truly independent and cannot be revoked without reasonable justification?", "f35c9f92-67ac-4772-9dab-6cf2ae32812f": "What are the legal requirements for providing notice when making a video recording of someone?", "d0edffef-580d-4890-a6dd-e08925fadd27": "How are companies and researchers improving automated systems to explain decisions that impact consumers?", "4a36f5cd-0f9d-42ba-bd8e-d0eaf0af2d52": "How do advertisement delivery systems reinforce racial and gender stereotypes?", "1c6de01d-b59d-4421-9339-0e501b4fd2b9": "What are the issues faced by transgender travelers with TSA body scanners at airport checkpoints?", "155db437-082c-44f4-8751-960146c3512c": "What are the five principles outlined in the Blueprint for an AI Bill of Rights?", "95cae333-a114-41e8-98f5-10619377f6bf": "How can organizations apply the Blueprint for an AI Bill of Rights to protect civil rights and privacy?", "077e8ee5-5768-4967-b8ed-891c6cc0085d": "What are the benefits of having a human fallback mechanism in automated systems?", "8edf6c51-407d-478c-832a-ef103ea3709e": "How do automated signature matching systems impact voters with mental or physical disabilities?", "7058b177-27f4-4d6b-a478-176ead46f325": "What are the best practices for documenting the sources and types of training data in AI models?", "1e48abdd-a664-4c7a-8f19-151ca61e5006": "How can user feedback be effectively integrated into system updates to address problematic content?", "e5aba341-abc2-4965-a224-fa10823f4d2f": "What is the two-part test used in the AI Bill of Rights framework to determine which systems are in scope?", "23c3711f-c55b-49e5-9936-22d6bfc010af": "How does the AI Bill of Rights framework ensure that automated systems do not negatively impact the American public's rights and access to critical resources?", "08a12dd0-5dd7-4f87-8913-d86a9cc2c8b7": "What are adversarial role-playing exercises and how do they help in identifying failure modes in GAI systems?", "7d2b3bbe-6d0b-470d-b85d-d0c636ac4354": "How can profiling threats and negative impacts improve the security of GAI systems?", "c385b92d-1c01-48ae-be4c-f6b42b5e6af6": "What are the potential negative impacts of school surveillance on students via laptops?", "b4286477-40f0-46b8-bba8-4fe204b0dafa": "How does \"Bossware\" affect the health of employees according to the Center for Democracy & Technology report?", "6c98dd15-2a73-4c66-8a6a-c578c67a2434": "How can employers ensure their use of AI in hiring complies with the Americans with Disabilities Act (ADA)?", "00ab3a02-dffb-482b-ad10-3cab6ad77520": "What are the potential risks of using healthcare algorithms that rely on past medical costs to predict future needs?", "510ed741-6a36-4d13-a7dc-6a42262136be": "What are some effective context-based measures to identify new impacts of GAI systems?", "8f74dbe1-c3ed-48ca-9635-d701d26e829a": "How can regular engagements with AI Actors help in evaluating unanticipated impacts of GAI systems?", "3809c393-b89e-494c-b529-c65e601c1544": "What are acceptable use policies for GAI interfaces and how do they determine the types of queries GAI applications should refuse to respond to?", "edb9b7b1-11c1-421c-a07f-7abe3d6e7c21": "How can organizations establish effective user feedback mechanisms for GAI systems, and what should these mechanisms include?", "5a48e740-85f0-48c7-b0c7-6247c384f052": "How often should adversarial testing be conducted to effectively map and measure GAI risks?", "d6567db0-b18c-4dcb-b80c-146f2047bc13": "What are the benefits of evaluating GAI system performance in real-world scenarios compared to controlled testing environments?", "59a37c01-7bac-4f9d-980f-48f5489e61e6": "What are the common statistics reported about who chooses the human alternative in automated systems?", "e24a71f0-8b86-461a-92bd-fa6cef7ca33b": "How often should reports on the accessibility, timeliness, and effectiveness of human consideration and fallback be made public?", "63dcc302-d64d-47f5-a304-a64d4d6642b4": "What are some examples of companies that have successfully implemented bias testing in their product quality assessment?", "9b9b4805-12cb-453d-a3f4-ddbb20679c39": "How are federal government agencies developing standards to prevent algorithmic discrimination?", "7b3d457a-d0bf-4b13-b59c-df184af98f08": "What are some common protections against unlawful surveillance and violations of privacy in both public and private sectors?", "9473baea-32cd-4147-a547-5d45b0daa757": "How can individuals ensure equitable access to education, housing, and employment opportunities?", "d4388801-831e-45e0-bf67-b67974027277": "What are the key principles outlined in the AI Bill of Rights?", "d4107956-2806-4098-a79e-e753cab1bf82": "How can the AI Bill of Rights be practically implemented in technical systems?", "829774bb-4770-46cf-9f1b-86f51e7b6679": "How can you ensure the data used in automated systems is of high quality and relevant to the task?", "79c355b3-3945-402d-9d15-e460689ba635": "What methods can be employed to measure and limit errors from data entry in automated systems?", "e558dbd7-ca81-4070-9777-49636694d674": "What are some reasons why certain risks cannot be measured quantitatively in AI systems?", "e1ce22f6-cad0-4bbe-87ae-5222158a4393": "How can organizations involve independent assessors and domain experts in the regular assessment of AI systems?", "ae84398b-1649-4cce-8fa2-6295c80f7ec9": "What are the risks associated with confabulated content in healthcare applications using GAI?", "648a7032-05c8-45c2-a7bb-2dca8fa9ffd0": "How can confabulated logic or citations from GAI systems mislead users?", "2b743770-5d66-4aa8-b9b4-c33adc78c1e3": "How can companies ethically use data to monitor employee performance without violating privacy?", "3b6f61ff-349d-4817-8c82-d064b9a71c86": "What are the legal implications of employers using surveillance data to intervene in employee discussions?", "f596fded-c16b-49cb-b400-734c65b185af": "What are the risks of using AI in high-stakes settings as highlighted by Pamela Wisniewski and Seny Kamara?", "1626655d-7f72-4d0a-9170-3abdc8ed86ec": "Why is it important to place trust in people rather than technologies when designing AI systems?", "cec6f35c-1b45-4d56-8c2f-aef7bc860a01": "How can organizations ensure that their demographic assessments are inclusive of all protected classifications?", "12aca964-2112-4b36-8a40-14ab1512ac75": "What are the best practices for separating demographic data used for disparity assessment from data used in automated systems?", "53a48063-f4fb-482f-bd70-36915ec63956": "What are some emerging technologies being used to improve social welfare systems?", "7fdbbfed-73aa-45a8-9f1c-58ec2c0f3912": "How can digital welfare systems impact life chances according to experts like Christiaan van Veen?", "0ed0fb9c-47c4-4c7c-a5ae-d7e3a35670a1": "What are some best practices for developers to ensure privacy by design in smartphone apps?", "88297ffa-b5ca-460c-81ed-a61975ab39ef": "How can developers make app permissions clear and use-specific for users?", "38409d77-4936-4266-a7f3-2d910d3bea91": "What are the privacy implications of using biometric identification technologies in New York schools?", "3d2d3a9e-a6a7-49f5-bdd8-5db95fc8b602": "What are the reporting requirements for employers who surveil employees during a labor dispute?", "ca685f83-ccd7-4a17-a31d-bfc648b58840": "What measures are included in the AI Bill of Rights to ensure automated systems are safe and effective?", "ce1fdffd-851d-463e-8f24-4596865b62dc": "How does the AI Bill of Rights propose to handle the risks and potential impacts of automated systems?", "1a82989c-3ead-4aea-9098-53d3dca7f9b7": "What are the potential downstream impacts of errors in third-party GAI components on system accuracy and robustness?", "a30ea710-3349-4357-8dcb-915f6c69f2da": "How can inaccuracies in test dataset labels affect the stability and robustness of GAI benchmarks?", "004b52ee-6a49-47d7-a4bd-77ec96fadc31": "What are the best practices for developing and updating GAI system incident response and recovery plans?", "a5ad1cc1-318a-4210-8838-22015d780344": "How can organizations ensure their response and recovery plans account for the entire GAI system value chain?", "f05e4729-18f1-4664-9f41-2ad997f9d726": "How can we assess the proportion of synthetic to non-synthetic training data in AI models?", "81c90ac3-caf0-4c9d-8e02-8c62d26a047e": "What are the best practices for documenting the environmental impacts of AI model development and deployment?", "0abf12fc-3e73-41e5-8594-5e2bb6ecdb24": "What are the primary considerations for organizations designing and developing GAI according to the GAI PWG consultation process?", "e3abf868-922a-42e7-8c5a-b1ff0a353d39": "How can governance principles and techniques be applied to manage risks in GAI systems?", "55c79cd5-dee3-4e43-b8a3-839028518379": "What are the key considerations for documenting the intended purposes and beneficial uses of an AI system?", "456333eb-689e-4896-b2d4-0cf136672c77": "How do internal vs external use and narrow vs broad application scope impact the identification of intended purposes for AI systems?", "8834b86c-b1b9-43d6-92e0-3c64ca09e854": "How can feedback from internal and external AI actors be used to assess the impact of AI-generated content?", "e84f1a90-e702-4594-84b8-5c5b67352195": "What are the benefits of using real-time auditing tools for tracking and validating the lineage and authenticity of AI-generated data?", "490b6ca7-059f-41fe-82ae-b8d2c3890cf1": "What are the main findings of Carlini et al (2024) regarding the vulnerabilities in production language models?", "59bed72b-bd80-47c3-bb57-08dd086ecf9d": "How does the study by Chandra et al (2023) propose to combat Chinese influence operations and disinformation?", "625e3e66-e1fc-4223-a201-e88b765f449e": "What is the role of the Electronic Privacy Information Center (EPIC) in AI policy and regulation?", "da4a10c9-db2a-45fa-bad5-b66ef842c023": "How does the Innocence Project utilize AI to support its mission?", "40ab1b55-bc53-4cae-8f7e-4657a5b2bdc2": "What is the role of the National Center for Missing & Exploited Children?", "46de7819-7250-4050-8bf9-4635a1a02f3e": "How does the New York Civil Liberties Union contribute to civil rights advocacy?", "6feae899-9900-454f-a64d-39e842af8c76": "How can AI tools be misused in the development of chemical or biological agents?", "36826afc-57e4-4d70-bc7e-4ca62e3e3e67": "What are the potential risks associated with the use of biological design tools (BDTs) in chemistry and biology?", "84440495-e768-4885-b78b-d8a0c17f3809": "How can expert AI red-teamers enhance the effectiveness of general public AI red-teamers?", "9c3a8107-d49c-4dc0-9f78-d71a506df892": "What are the benefits of using GAI-led red-teaming compared to human red-teamers alone?", "068d8bd2-9336-4e18-bd93-2199100e631f": "How can error ranges be calculated and included in explanations for decision-making systems?", "3138ca26-38b8-4e17-9b31-b38bc8a8eb4f": "What are the best practices for balancing usability and interface complexity when presenting decision-making information?", "095919bc-18fa-4316-b1e8-07572983b77b": "What are the potential benefits and drawbacks of using predictive policing in the criminal justice system?", "39406f17-a757-4201-91b5-284ba4ebbd39": "How can data-driven approaches be balanced with the need for community safety in criminal justice reform?", "2744b9cf-981d-42e5-aed3-bb8e5acb0b2e": "What are the reporting expectations for entities developing or using automated systems?", "798b53f4-f798-418a-abcd-6dd05f707c67": "How can the public access the Agency Inventories of AI Use Cases provided by the National Artificial Intelligence Initiative Office?", "d7fa2d65-26f8-4442-86f6-f1d6256e588a": "What are some effective methods for monitoring and assessing high-impact systems in qualitative user experience research?", "e50c31b3-bab1-4064-baa1-199c946d9789": "How can organizations ensure equity standards are maintained in algorithmic systems, and what steps should be taken if these standards are not met?", "644dcaa5-1731-43fe-b0f5-c6a4bc05564e": "What factors should be considered when updating or defining risk tiers for General Artificial Intelligence (GAI)?", "def43eb9-80b0-4ad2-9198-d84ecb89c720": "How can the psychological impacts of GAI, such as anthropomorphization and emotional entanglement, be mitigated?", "8495a23f-4bb7-47ac-8c54-58cf5675cdd7": "What are the best practices for establishing policies to manage risks related to rollover and fallback technologies in GAI systems?", "74ae51e9-63b3-48ce-9be7-4f88052d7bd6": "How can organizations ensure clear assignment of liability and responsibility in vendor contracts for GAI technologies?", "11cdd3ed-e09b-463d-9853-0be811073b75": "What are the best practices for ensuring the confidentiality of AI training data and model weights?", "9ca1ff0e-0cd9-4362-aca9-fd904077c845": "How can potential attack points in AI systems be identified and secured?", "8fe0054d-51ba-48c5-8cc5-259b2b96f535": "How can AI-powered cameras in delivery vans be improved to avoid incorrectly penalizing drivers?", "03b9f17b-0b61-401b-bc65-47d0655f31d8": "What are the common issues faced by companies using AI to monitor road safety habits of drivers?", "6d622041-fccf-4eb4-9a53-f7d7577856f8": "What are the differences in resource usage between AI training and inference?", "a1738003-3e17-48e7-86a2-1410bc0f1c07": "How can we verify the effectiveness of carbon capture programs for AI training?", "d15e0c10-378f-48a3-9a5c-be0c618106b4": "What protocols should be in place to ensure the safe deactivation of AI systems?", "7e7e2c28-ea80-4568-a71a-41966f9f117f": "What factors need to be considered when decommissioning AI systems to prevent data leakage and ensure security?", "57073541-fc8c-43cd-8b42-f9497eb501af": "What are the best practices for limiting access to sensitive data based on necessity and local control?", "92d9e36d-0fef-4b2e-b40d-ff2b800fcf10": "How should organizations report data security lapses or breaches involving sensitive data?", "6f7aa060-c19a-4614-83d2-134828a7e956": "What is the purpose of the email address ai-equity@ostpeopgov created by OSTP?", "6b95bc28-dbb4-408f-8c5b-f5b37073b6fd": "Where can I find the full responses to the OSTP's Request For Information (RFI) on biometric technologies?", "4776eaa1-b6f0-440c-a6be-923bbf49687d": "What are the practical steps to implement ethical principles in technology?", "acf74d86-1184-4092-8a1d-3ca58f5fe97a": "How can risk management be integrated into technological innovation to protect people from harm?", "2c1b02c6-1919-49ea-beff-165567d20b47": "What are the key capabilities needed for automated systems to help users make consent, access, and control decisions in a complex data ecosystem?", "2d15dfed-c66d-4fac-89dd-3aded02ec63e": "How can independent evaluations of data policies help ensure data privacy and user control in automated systems?", "e71beb7c-7564-4f2c-83f7-ec9bb3134847": "How can the rate of implementing recommendations from security checks and incidents be measured effectively?", "6c079fa0-60c3-4c8d-826a-2816c65d3ea0": "What are the best practices for performing AI red-teaming to assess resilience against various types of attacks?", "543e9bfb-b5f4-4247-89c8-41e0e7fb11a9": "What are the legal and regulatory requirements for reporting GAI incidents under HIPAA?", "8613b055-c817-4a59-84cf-1ae29a7c2269": "How does the NHTSA's 2022 autonomous vehicle crash reporting requirements impact AI deployment and monitoring?", "ce252388-c4d9-4968-aadf-218b47f609a5": "How do you document the justification for each data attribute in an automated system?", "f46088d7-1004-41cb-87c5-8a2b0bcdef59": "What are the best practices for ensuring that the use of high-dimensional data attributes does not violate applicable laws?", "b5f49997-5049-4865-9b5b-c18d880e2baf": "How can organizations adjust their governance regimes to effectively manage the risks associated with generative AI systems?", "eeb5acfd-3be2-4488-b45e-e0979bd5c855": "What are the key considerations for third-party governance across the AI value chain when dealing with generative AI?"}

Tasks/Task 5/task5-assessing-performance.ipynb

@@ -0,0 +1,709 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "!pip install -qU langsmith langchain-core langchain-community langchain-openai langchain-qdrant langchain_experimental pymupdf ragas"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import getpass\n",
+    "from uuid import uuid4\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_TRACING_V2\"] = \"true\"\n",
+    "os.environ[\"LANGCHAIN_API_KEY\"] = getpass.getpass(\"LangChain API Key:\")\n",
+    "\n",
+    "os.environ[\"LANGCHAIN_PROJECT\"] = \"AIM-SDG-MidTerm - AI Safety\"\n",
+    "os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")\n",
+    "\n",
+    "os.environ[\"QDRANT_API_KEY\"] = getpass.getpass(\"Enter Your Qdrant API Key: \")"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Synthetic data generation using Ragas framework"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We will generate set of synthetic data for evaluating different opetions\n",
+    "1. Evaluating Embedding model\n",
+    "2. Evaluating Chunking Strategies"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_experimental.text_splitter import SemanticChunker\n",
+    "\n",
+    "from enum import Enum\n",
+    "from typing import List\n",
+    "from langchain_community.document_loaders import PyMuPDFLoader\n",
+    "from langchain_core.documents import Document\n",
+    "import asyncio\n",
+    "\n",
+    "class PDFLoaderWrapper():\n",
+    "    class LoaderType(str, Enum):\n",
+    "        PYMUPDF = \"pymupdf\"\n",
+    "\n",
+    "    def __init__(self, file_path: str | List[str] , loader_type: LoaderType = LoaderType.PYMUPDF):\n",
+    "        self.file_path = file_path if isinstance(file_path, list) else [file_path]\n",
+    "        self.loader_type = loader_type\n",
+    "\n",
+    "    async def aload(self) -> List[Document]:\n",
+    "        all_docs = []\n",
+    "        for file_path in self.file_path:\n",
+    "            if self.loader_type == self.LoaderType.PYMUPDF:\n",
+    "                try:\n",
+    "                    loader = PyMuPDFLoader(file_path)\n",
+    "                    docs = await loader.aload()\n",
+    "                    all_docs.extend(docs)\n",
+    "                except Exception as e:\n",
+    "                    print(f\"Error loading file {file_path}: {e}\")\n",
+    "                    continue\n",
+    "        return all_docs\n",
+    "\n",
+    "BOR_FILE_PATH = \"https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf\"\n",
+    "NIST_FILE_PATH = \"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf\"\n",
+    "documents = [\n",
+    "    BOR_FILE_PATH,\n",
+    "    NIST_FILE_PATH\n",
+    "]\n",
+    "\n",
+    "pdf_loader = PDFLoaderWrapper(\n",
+    "    documents, PDFLoaderWrapper.LoaderType.PYMUPDF\n",
+    ")\n",
+    "documents = await pdf_loader.aload()\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from ragas.testset.generator import TestsetGenerator\n",
+    "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
+    "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
+    "from ragas.testset.docstore import Document, DocumentStore,InMemoryDocumentStore\n",
+    "from langchain_experimental.text_splitter import SemanticChunker,RecursiveCharacterTextSplitter\n",
+    "from langchain_huggingface import HuggingFaceEmbeddings, HuggingFacePipeline\n",
+    "from ragas.testset.extractor import KeyphraseExtractor\n",
+    "\n",
+    "print (\"Packages import complete\")\n",
+    "print (\"Getting the Embedding model from Huggingface\")\n",
+    "# Using best performing embedding model from hugging face to generate quality dataset.\n",
+    "# Need GPU\n",
+    "model_name = \"Snowflake/snowflake-arctic-embed-l\"\n",
+    "embedding_model = HuggingFaceEmbeddings(model_name=model_name)\n",
+    "print (\"Embedding model loaded\")\n",
+    "\n",
+    "print (\"Splitting the documents into semantic chunks\")\n",
+    "text_splitter = RecursiveCharacterTextSplitter(\n",
+    "    chunk_size = 1024,\n",
+    "    chunk_overlap = 100,\n",
+    "    length_function = len,\n",
+    ")\n",
+    "chunked_docs = text_splitter.split_documents(documents)\n",
+    "\n",
+    "print (\"Creating the document store for ragas and loading LLM models\")\n",
+    "generator_llm = ChatOpenAI(model=\"gpt-4o\")\n",
+    "critic_llm = ChatOpenAI(model=\"gpt-4o\")\n",
+    "\n",
+    "keyphrase_extractor = KeyphraseExtractor(llm=generator_llm)\n",
+    "docstore = InMemoryDocumentStore(splitter=text_splitter,extractor=keyphrase_extractor, embeddings=embedding_model)\n",
+    "\n",
+    "print (\"Creating the testset generator\")\n",
+    "generator = TestsetGenerator.from_langchain( # Default uses TokenTextSplitter\n",
+    "    generator_llm=generator_llm,\n",
+    "    critic_llm=critic_llm,\n",
+    "    embeddings=embedding_model,\n",
+    "    docstore=docstore # Document store uses SemenaticChunker\n",
+    ")\n",
+    "\n",
+    "distributions = {\n",
+    "    simple: 0.5,\n",
+    "    multi_context: 0.3,\n",
+    "    reasoning: 0.2\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "test_size = 50\n",
+    "\n",
+    "testset = generator.generate_with_langchain_docs(\n",
+    "    documents, \n",
+    "    test_size, \n",
+    "    distributions, \n",
+    "    with_debugging_logs=True\n",
+    ") # Default  RunConfig(max_retries=15, max_wait=90)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "testset_df = testset.to_pandas()\n",
+    "testset_df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "testset_df.to_csv('task5-ai-safety-sdg.csv', index=False)\n",
+    "test_questions = testset_df[\"question\"].values.tolist()\n",
+    "test_groundtruths = testset_df[\"ground_truth\"].values.tolist()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create Rag chain to generate answers for above questions in the dataset"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "> Note that we are usig Qdrant cloud where the pdf document is processed and saved for us to consume. For the RAG pipeline we use the same embedding model originally used to populate the Qdrant vectorstore."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_qdrant import QdrantVectorStore\n",
+    "from langchain_core.documents import Document\n",
+    "from qdrant_client import QdrantClient\n",
+    "from qdrant_client.http.models import Distance, VectorParams\n",
+    "\n",
+    "dimension = 1024\n",
+    "collection_name = \"ai-safety-sr-arctic-embed-l-recursive\"\n",
+    "qdrant_server = \"https://500cb0e8-ea08-4662-b4f2-3eca11e635da.europe-west3-0.gcp.cloud.qdrant.io:6333\"\n",
+    "qdrant_client = QdrantClient(url=qdrant_server,api_key=os.environ[\"QDRANT_API_KEY\"])\n",
+    "qdrant_client.create_collection(\n",
+    "    collection_name=collection_name,\n",
+    "    vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),\n",
+    ")\n",
+    "\n",
+    "vector_store = QdrantVectorStore(\n",
+    "    client=qdrant_client,\n",
+    "    collection_name=collection_name,\n",
+    "    embedding=embedding_model,\n",
+    ")\n",
+    "\n",
+    "retriever = vector_store.as_retriever()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "\n",
+    "RAG_PROMPT = \"\"\"\\\n",
+    "Given a provided context and question, you must answer the question based only on context.\n",
+    "\n",
+    "If you cannot answer the question based on the context - you must say \"I don't know\".\n",
+    "\n",
+    "Context: {context}\n",
+    "Question: {question}\n",
+    "\"\"\"\n",
+    "\n",
+    "rag_prompt = ChatPromptTemplate.from_template(RAG_PROMPT)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain_openai import ChatOpenAI\n",
+    "\n",
+    "# Using the same model used in the app.\n",
+    "chat_model_name = \"gpt-4o\"\n",
+    "llm = ChatOpenAI(model=chat_model_name)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from operator import itemgetter\n",
+    "from langchain_core.runnables import RunnablePassthrough, RunnableParallel\n",
+    "from langchain.schema import StrOutputParser\n",
+    "\n",
+    "rag_chain_generate_anwsers = (\n",
+    "    {\"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"question\")}\n",
+    "    | rag_prompt | llm | StrOutputParser()\n",
+    ")"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "rag_chain_generate_anwsers.invoke({\"question\" : \"What steps can organizations take to minimize bias in AI models?\"})"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Create Rag Chain with config "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "We are going to replicate the exact implementation used in the hosted RAG app but with different configuration to evaluate and compare."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# Utility function to create a rag chain with config\n",
+    "from langchain_experimental.text_splitter import SemanticChunker\n",
+    "from enum import Enum\n",
+    "from typing import List\n",
+    "from langchain_community.document_loaders import PyMuPDFLoader\n",
+    "from langchain_core.documents import Document\n",
+    "import asyncio\n",
+    "from langchain_qdrant import QdrantVectorStore\n",
+    "from langchain_core.documents import Document\n",
+    "from qdrant_client import QdrantClient\n",
+    "from qdrant_client.http.models import Distance, VectorParams\n",
+    "from langchain.retrievers.contextual_compression import ContextualCompressionRetriever\n",
+    "from langchain.retrievers.document_compressors import LLMChainExtractor\n",
+    "from langchain_openai import ChatOpenAI\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "from operator import itemgetter\n",
+    "from langchain_core.runnables import RunnablePassthrough, RunnableParallel\n",
+    "from langchain.schema import StrOutputParser\n",
+    "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
+    "from langchain.prompts import MessagesPlaceholder\n",
+    "from langchain.prompts import ChatPromptTemplate\n",
+    "from langchain.chains.history_aware_retriever import create_history_aware_retriever\n",
+    "from langchain.chains.retrieval import create_retrieval_chain\n",
+    "from langchain_core.runnables.history import RunnableWithMessageHistory\n",
+    "from langchain_core.chat_history import BaseChatMessageHistory\n",
+    "from langchain_community.chat_message_histories import ChatMessageHistory\n",
+    "\n",
+    "BOR_FILE_PATH = \"https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf\"\n",
+    "NIST_FILE_PATH = \"https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf\"\n",
+    "documents_to_preload = [\n",
+    "    BOR_FILE_PATH,\n",
+    "    NIST_FILE_PATH\n",
+    "]\n",
+    "store = {}\n",
+    "\n",
+    "class PDFLoaderWrapper():\n",
+    "    class LoaderType(str, Enum):\n",
+    "        PYMUPDF = \"pymupdf\"\n",
+    "\n",
+    "    def __init__(self, file_path: str | List[str] , loader_type: LoaderType = LoaderType.PYMUPDF):\n",
+    "        self.file_path = file_path if isinstance(file_path, list) else [file_path]\n",
+    "        self.loader_type = loader_type\n",
+    "\n",
+    "    async def aload(self) -> List[Document]:\n",
+    "        all_docs = []\n",
+    "        for file_path in self.file_path:\n",
+    "            if self.loader_type == self.LoaderType.PYMUPDF:\n",
+    "                try:\n",
+    "                    loader = PyMuPDFLoader(file_path)\n",
+    "                    docs = await loader.aload()\n",
+    "                    all_docs.extend(docs)\n",
+    "                except Exception as e:\n",
+    "                    print(f\"Error loading file {file_path}: {e}\")\n",
+    "                    continue\n",
+    "        return all_docs\n",
+    "\n",
+    "async def get_contextual_compressed_retriever(retriver):\n",
+    "\n",
+    "    base_retriever = retriver\n",
+    "    compressor_llm = ChatOpenAI(temperature=0, model_name=\"gpt-4o\", max_tokens=1500)\n",
+    "    compressor = LLMChainExtractor.from_llm(compressor_llm)\n",
+    "\n",
+    "    #Combine the retriever with the compressor\n",
+    "    compression_retriever = ContextualCompressionRetriever(\n",
+    "        base_compressor=compressor,\n",
+    "        base_retriever=base_retriever\n",
+    "    )\n",
+    "    return compression_retriever\n",
+    "\n",
+    "def create_history_aware_retriever_self(chat_model, retriever):\n",
+    "    contextualize_q_system_prompt = (\n",
+    "        \"Given a chat history and the latest user question which might reference context in the chat history, \"\n",
+    "        \"formulate a standalone question which can be understood without the chat history. Do NOT answer the question, \"\n",
+    "        \"just reformulate it if needed and otherwise return it as is.\"\n",
+    "    )\n",
+    "    contextualize_q_prompt = ChatPromptTemplate.from_messages(\n",
+    "        [\n",
+    "            (\"system\", contextualize_q_system_prompt),\n",
+    "            MessagesPlaceholder(\"chat_history\"),\n",
+    "            (\"human\", \"{input}\"),\n",
+    "        ]\n",
+    "    )\n",
+    "    return create_history_aware_retriever(chat_model, retriever, contextualize_q_prompt)\n",
+    "\n",
+    "def create_qa_chain(chat_model):\n",
+    "    qa_system_prompt = (\n",
+    "        \"You are an helpful assistant named 'Shield' and your task is to answer any questions related to AI Safety for the given context.\"\n",
+    "        \"Use the following pieces of retrieved context to answer the question.\"\n",
+    "        # \"If any questions asked outside AI Safety context, just say that you are a specialist in AI Safety and can't answer that.\"\n",
+    "        # f\"When introducing you, just say that you are an AI assistant powered by embedding model {embedding_model_name} and chat model {chat_model_name} and your knowledge is limited to 'Blueprint for an AI Bill of Rights' and 'NIST AI Standards' documents.\"\n",
+    "        \"If you don't know the answer, just say that you don't know.\\n\\n\"\n",
+    "        \"{context}\"\n",
+    "    )\n",
+    "    qa_prompt = ChatPromptTemplate.from_messages(\n",
+    "        [\n",
+    "            (\"system\", qa_system_prompt),\n",
+    "            MessagesPlaceholder(\"chat_history\"),\n",
+    "            (\"human\", \"{input}\"),\n",
+    "        ]\n",
+    "    )\n",
+    "    return create_stuff_documents_chain(chat_model, qa_prompt)\n",
+    "\n",
+    "def create_conversational_rag_chain(chat_model, retriever):\n",
+    "    history_aware_retriever = create_history_aware_retriever_self(chat_model, retriever)\n",
+    "    question_answer_chain = create_qa_chain(chat_model)\n",
+    "    return create_retrieval_chain(history_aware_retriever, question_answer_chain)\n",
+    "\n",
+    "def get_session_history(session_id: str) -> BaseChatMessageHistory:\n",
+    "    if session_id not in store:\n",
+    "        store[session_id] = ChatMessageHistory()\n",
+    "    return store[session_id]\n",
+    "\n",
+    "\n",
+    "pdf_loader = PDFLoaderWrapper(\n",
+    "    documents_to_preload, PDFLoaderWrapper.LoaderType.PYMUPDF\n",
+    ")\n",
+    "documents = await pdf_loader.aload()\n",
+    "\n",
+    "async def create_history_aware_rag_chain_with_params(huggingface_embedding,text_splitter,collection_name,compress:bool=False,conversational:bool =False):\n",
+    "\n",
+    "    chunked_docs = text_splitter.split_documents(documents)\n",
+    "    dimension = 1024\n",
+    "    qdrant_server = os.environ[\"QDRANT_URL\"]\n",
+    "    qdrant_client = QdrantClient(url=qdrant_server,api_key=os.environ[\"QDRANT_API_KEY\"])\n",
+    "\n",
+    "    # Below fails if collection already exists so make sure to delete the collection first\n",
+    "    qdrant_client.create_collection(\n",
+    "        collection_name=collection_name,\n",
+    "        vectors_config=VectorParams(size=dimension, distance=Distance.COSINE),\n",
+    "    )\n",
+    "\n",
+    "    vector_store = QdrantVectorStore(\n",
+    "        client=qdrant_client,\n",
+    "        collection_name=collection_name,\n",
+    "        embedding=huggingface_embedding,\n",
+    "    )\n",
+    "    vector_store.add_documents(chunked_docs)\n",
+    "\n",
+    "    retriever = vector_store.as_retriever(search_type=\"similarity_score_threshold\",\n",
+    "                    search_kwargs={'k':10,'score_threshold': 0.8})\n",
+    "    \n",
+    "     # Using the same model used in the app.\n",
+    "    chat_model_name = \"gpt-4o\"\n",
+    "    llm = ChatOpenAI(model=chat_model_name,temperature=0)  \n",
+    "\n",
+    "    if compress:\n",
+    "        contextual_compressed_retriever = await get_contextual_compressed_retriever(retriever)\n",
+    "    \n",
+    "    if conversational:\n",
+    "        history_ai_safety_rag_chain = create_conversational_rag_chain(\n",
+    "            llm, \n",
+    "            contextual_compressed_retriever if contextual_compressed_retriever else retriever)\n",
+    "\n",
+    "        conversational_rag_chain = RunnableWithMessageHistory(\n",
+    "            history_ai_safety_rag_chain,\n",
+    "            get_session_history,\n",
+    "            input_messages_key=\"input\",\n",
+    "            history_messages_key=\"chat_history\",\n",
+    "            output_messages_key=\"answer\",\n",
+    "        )\n",
+    "    else:\n",
+    "        RAG_PROMPT = \"\"\"\\\n",
+    "        Given a provided context and question, you must answer the question based only on context.\n",
+    "\n",
+    "        If you cannot answer the question based on the context - you must say \"I don't know\".\n",
+    "\n",
+    "        Context: {context}\n",
+    "        Question: {question}\n",
+    "        \"\"\"\n",
+    "\n",
+    "        rag_prompt = ChatPromptTemplate.from_template(RAG_PROMPT)\n",
+    "        ai_safety_rag_chain = (\n",
+    "            {\"context\": itemgetter(\"question\") | retriever, \"question\": itemgetter(\"input\")}\n",
+    "            | rag_prompt | llm | StrOutputParser()\n",
+    "        )\n",
+    "\n",
+    "    ret = contextual_compressed_retriever if contextual_compressed_retriever else retriever\n",
+    "    chain = conversational_rag_chain if conversational_rag_chain else ai_safety_rag_chain\n",
+    "    return chain, ret\n",
+    "\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# RAGAS Evaluation for Embedding Model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
+    "# Load the embedding models\n",
+    "base_embedding_model = HuggingFaceEmbeddings(model_name=\"Snowflake/snowflake-arctic-embed-l\")\n",
+    "fine_tuned_embedding_model = HuggingFaceEmbeddings(model_name=\"<need to updaet one ft finished>\")\n",
+    "\n",
+    "# Common splitter to keep variables minimum\n",
+    "recursive_text_splitter = RecursiveCharacterTextSplitter(\n",
+    "    chunk_size = 1024,\n",
+    "    chunk_overlap = 100,\n",
+    "    length_function = len,\n",
+    ")\n",
+    "recursive_chunked_docs = recursive_text_splitter.split_documents(documents)\n",
+    "\n",
+    "# Create two rag chaings with different embeddings\n",
+    "rag_chain_base, retriever_base = await create_history_aware_rag_chain_with_params(\n",
+    "    base_embedding_model,\n",
+    "    recursive_text_splitter,\n",
+    "    \"aichatty-snowflake-arctic-embed-l-recursive-base\"\n",
+    ")\n",
+    "\n",
+    "rag_chain_fine_tuned, retriever_fine_tuned = await create_history_aware_rag_chain_with_params(\n",
+    "    fine_tuned_embedding_model,\n",
+    "    recursive_text_splitter,\n",
+    "    \"aichatty-snowflake-arctic-embed-l-recursive-ft\"\n",
+    ")\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# First Base model chain\n",
+    "\n",
+    "from datasets import Dataset\n",
+    "import time\n",
+    "import uuid\n",
+    "from ragas import evaluate\n",
+    "from ragas.metrics import (\n",
+    "    faithfulness,\n",
+    "    answer_relevancy,\n",
+    "    answer_correctness,\n",
+    "    context_recall,\n",
+    "    context_precision,\n",
+    ")\n",
+    "\n",
+    "answers = []\n",
+    "contexts = []\n",
+    "\n",
+    "for question in test_questions:\n",
+    "  store = {}\n",
+    "  session_id = str(uuid.uuid4())\n",
+    "\n",
+    "  response = rag_chain_base.invoke({\"input\" : question},  config={\"configurable\": {\"session_id\": session_id}})\n",
+    "  answers.append(response[\"answer\"])\n",
+    "  contexts.append([context.page_content for context in response[\"context\"]])\n",
+    "\n",
+    "base_chain_response_dataset = Dataset.from_dict({\n",
+    "    \"question\" : test_questions,\n",
+    "    \"answer\" : answers,\n",
+    "    \"contexts\" : contexts,\n",
+    "    \"ground_truth\" : test_groundtruths\n",
+    "})\n",
+    "\n",
+    "metrics = [\n",
+    "    faithfulness,\n",
+    "    answer_relevancy,\n",
+    "    context_recall,\n",
+    "    context_precision,\n",
+    "    answer_correctness,\n",
+    "]\n",
+    "base_chain_eval_results = evaluate(base_chain_response_dataset, metrics)\n",
+    "\n",
+    "base_chain_eval_results_df = base_chain_eval_results.to_pandas()\n",
+    "base_chain_eval_results_df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# For Fine Tuned model chain\n",
+    "\n",
+    "answers = []\n",
+    "contexts = []\n",
+    "\n",
+    "for question in test_questions:\n",
+    "  store = {}\n",
+    "  session_id = str(uuid.uuid4())\n",
+    "\n",
+    "  response = rag_chain_fine_tuned.invoke({\"input\" : question},  config={\"configurable\": {\"session_id\": session_id}})\n",
+    "  answers.append(response[\"answer\"])\n",
+    "  contexts.append([context.page_content for context in response[\"context\"]])\n",
+    "\n",
+    "ft_chain_response_dataset = Dataset.from_dict({\n",
+    "    \"question\" : test_questions,\n",
+    "    \"answer\" : answers,\n",
+    "    \"contexts\" : contexts,\n",
+    "    \"ground_truth\" : test_groundtruths\n",
+    "})\n",
+    "\n",
+    "ft_chain_eval_results = evaluate(ft_chain_response_dataset, metrics)\n",
+    "\n",
+    "ft_chain_eval_results_df = ft_chain_eval_results.to_pandas()\n",
+    "ft_chain_eval_results_df"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import pandas as pd\n",
+    "import matplotlib.pyplot as plt\n",
+    "\n",
+    "# Load your dataframes here\n",
+    "\n",
+    "# Merge on the 'question' column or another common identifier\n",
+    "merged_df = pd.merge(base_chain_eval_results_df, ft_chain_eval_results_df, on='question', suffixes=('_base', '_finetuned'))\n",
+    "\n",
+    "# Now, let's calculate the improvement in relevant metrics, e.g., answer_correctness\n",
+    "merged_df['improvement_answer_correctness'] = merged_df['answer_correctness_finetuned'] - merged_df['answer_correctness_base']\n",
+    "\n",
+    "# Plotting the comparison of correctness between the two models\n",
+    "plt.figure(figsize=(10, 6))\n",
+    "plt.plot(merged_df['question'], merged_df['answer_correctness_base'], label='Base Model', marker='o')\n",
+    "plt.plot(merged_df['question'], merged_df['answer_correctness_finetuned'], label='Fine-tuned Model', marker='x')\n",
+    "plt.xlabel('Questions')\n",
+    "plt.ylabel('Answer Correctness')\n",
+    "plt.title('Comparison of Base Model and Fine-tuned Model on Answer Correctness')\n",
+    "plt.xticks(rotation=90)\n",
+    "plt.legend()\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n",
+    "\n",
+    "# Plotting the improvement\n",
+    "plt.figure(figsize=(10, 6))\n",
+    "plt.bar(merged_df['question'], merged_df['improvement_answer_correctness'], color='green')\n",
+    "plt.xlabel('Questions')\n",
+    "plt.ylabel('Improvement in Answer Correctness')\n",
+    "plt.title('Improvement in Answer Correctness after Fine-tuning')\n",
+    "plt.xticks(rotation=90)\n",
+    "plt.tight_layout()\n",
+    "plt.show()\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# RAGAS Evaluation for chunking strategy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+    "\n",
+    "# Using fine_tuned_embedding_model from before\n",
+    "\n",
+    "# First Splitter RecursiveCharacterTextSplitter\n",
+    "recursive_text_splitter = RecursiveCharacterTextSplitter(\n",
+    "    chunk_size = 1024,\n",
+    "    chunk_overlap = 100,\n",
+    "    length_function = len,\n",
+    ")\n",
+    "recursive_chunked_docs = recursive_text_splitter.split_documents(documents)\n",
+    "\n",
+    "# Create two rag chaings with different embeddings\n",
+    "conversational_rag_chain_base, contextual_compressed_retriever_base, retriever_base = await create_history_aware_rag_chain_with_params(\n",
+    "    base_embedding_model,recursive_text_splitter,\"aichatty-snowflake-arctic-embed-l-recursive-base\"\n",
+    ")\n",
+    "\n",
+    "conversational_rag_chain_fine_tuned, contextual_compressed_retriever_fine_tuned, retriever_fine_tuned = await create_history_aware_rag_chain_with_params(\n",
+    "    fine_tuned_embedding_model,recursive_text_splitter,\"aichatty-snowflake-arctic-embed-l-recursive-ft\"\n",
+    ")\n"
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": ".venv",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.11.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

Tasks/Task 6/task6.md

@@ -0,0 +1,25 @@
+### 1. **What is the story that you will give to the CEO to tell the whole company at the launch next month?**
+
+The story for the CEO should emphasize how the company's investment in AI technology is yielding tangible results and preparing the organization for the future of AI governance. 
+
+
+**CEO Narrative:**
+
+"Over the past months, our dedicated team of engineers and AI specialists have worked on a groundbreaking Retrieval-Augmented Generation (RAG) application. We’ve tested the system rigorously across 50 internal stakeholders, collecting feedback and refining the performance metrics based on cutting-edge frameworks like RAGAS. This initiative places us at the forefront of ethical AI adoption, ensuring our systems align with upcoming regulatory frameworks such as the 2024 NIST AI Risk Management Framework and the AI Bill of Rights. 
+
+We’ve created a scalable and adaptable AI system that not only answers complex questions related to AI safety but also ensures that our internal teams have the tools to address customer concerns, especially in today's rapidly changing AI landscape. By next month, we’ll be ready to roll out the solution across the enterprise, showing our commitment to innovation, security, and accountability."
+
+
+### 2. **How might you incorporate relevant white-house briefing information into future versions?**
+
+In future versions of the RAG application, it is critical to keep it aligned with national regulations like the executive order on Safe, Secure, and Trustworthy AI. Here's how:
+
+- **Compliance & Transparency:** Ensure that future updates integrate guidelines from the White House’s executive orders on AI governance. This can be achieved by:
+   - **Embedding legal updates** into the knowledge base of the RAG system. This will allow the system to reflect the latest compliance requirements and help internal stakeholders understand how AI aligns with regulations.
+   - Implement an **alert system** within the RAG tool to notify users whenever regulatory changes impact AI practices.
+
+- **Periodic Review:** Set a quarterly update schedule where AI regulations, such as the 270-day update mentioned, are reviewed, and the RAG application is updated accordingly. This ensures that the company stays proactive regarding AI safety and governance.
+
+At the moment these guidelines are loaded into the system offline and in the future we will be rolling out features that keep the system constantly updated on any new regislation. 
+
+This approach will position the company not only as an industry leader but also as a responsible organization that prioritizes safe and ethical AI use.

Tasks/deliverables.md

@@ -0,0 +1,153 @@
+## Task 1 - Dealing with the data
+
+***Deliverable 1*** Describe the default chunking strategy that you will use.
+
+We started of using `SemanticChunker` based on our understand, this would provide quality retrieval of the knowledge to enable LLM generate the response for a given question. 
+
+***Deliverable 2*** Articulate a chunking strategy that you would also like to test out.
+
+I would like to explore `RecursiveCharacterTextSplitter` with some advanced RAG techniques like Contextual Compression or Parenet Document retriever.
+
+***Deliverable 3*** Describe how and why you made these decisions
+
+From the past experience and also in doing this mid term project, I understood one thing "Start with simple changes and iterate over period of time. Any small change in the RAG setup drasticly impact the performance and quality of the output." So wanted to be simple, smart and easy to change.
+
+> Since loading huge pdfs and populating vector store is time consuming I ended up in doing pre processing where the vector store is kept ready to be consumed. Refer the notebook in folder ![Task1](/ai-safety-chatty/Tasks/Task%201/)
+> To decide on chunking strategy I worked on seperate notebook to try different option. Refer ![Task1](/ai-safety-chatty/Tasks/Task%201/settingup-vectorstore-chunking-strategy.ipynb)
+> Refer the ![Task1-Detailed](/ai-safety-chatty/Tasks/Task%201/Task1.md) For detailed workout of this task.
+
+## Task 2 - Building a Quick End-to-End Prototype
+
+***Deliverable 1*** Build a prototype and deploy to a Hugging Face Space, and include the public URL link to your space  create a short (< 2 min) loom video demonstrating some initial testing inputs and outputs.
+
+HF URL: 
+Loom Video:
+
+***Deliverable 2*** How did you choose your stack, and why did you select each tool the way you did?
+
+1. **Qdrant**:  
+   - **Reason for Selection**: Provides efficient, scalable, and fast vector search for embedding retrieval, crucial for the RAG framework.
+
+2. **PyMuPDFLoader**:  
+   - **Reason for Selection**: Lightweight and fast PDF parsing, ideal for loading structured and dense documents like the AI Bill of Rights.
+
+3. **RecursiveCharacterTextSplitter**:  
+   - **Reason for Selection**: Allows flexible chunking while preserving the semantic context, improving retrieval precision.
+
+4. **SemanticChunker**:  
+   - **Reason for Selection**: Enables semantically rich text chunking, leading to better coherence and improved retrieval results.
+
+5. **Snowflake-Arctic-Embed-L Embedding Model**:  
+   - **Reason for Selection**: A smaller, efficient model providing a good balance between speed and accuracy for embedding text chunks in RAG systems. One of the highly ranked model as per MTEB.
+
+6. **Context Enrichment and Contextual Compression**:  
+   - **Reason for Selection**: Enhances the retrieval process by providing more targeted, concise, and context-rich answers.
+  
+7. **Tracing**: Used LangSmith for tracing which natively works with other frameworks and helps understand the issues we have in the RAG chain.
+   
+8. **RAGAS**: Used to evaluate our RAG system with different configurations. This improve the performance of the RAG application.
+
+This stack was designed with a focus on balancing performance, scalability, and precision to build an effective Retrieval-Augmented Generation (RAG) application.
+
+Each tool in this stack was chosen to ensure **speed**, **scalability**, and **accuracy** while dealing with structured and unstructured documents. By balancing performance with precision (e.g., fast document loading via PyMuPDFLoader, efficient chunking strategies, and a small but powerful embedding model), this stack provides a robust framework for building ethical and useful AI applications.
+
+> Refer additional notes in ![Task2](/ai-safety-chatty/Tasks/Task%202/Task2.md)
+
+## Task 3 - Creating a Golden Test DataSet
+
+***Deliverable 1*** Assess your pipeline using the RAGAS framework including key metrics faithfulness, answer relevancy, context precision, and context recall.  Provide a table of your output results.
+
+![Result](/ai-safety-chatty/Tasks/Task%203/task3-del1.png)
+![Condensd Result](/ai-safety-chatty/Tasks/Task%203/task3-del11.png)
+
+***Deliverable 2*** What conclusions can you draw about performance and effectiveness of your pipeline with this information?
+
+Some observations from the results
+
+- **Faithfulness**: Mostly high faithfulness scores, indicating that the generated answers are generally true to the source material. But, there are some low score (e.g., 0.233333) which shows that the model may occasionally provide unfaithful or incomplete answers.
+  
+- **Answer Relevancy**: The model seems to perform well in answer relevancy, with most scores being near 1. This suggests that even when faithfulness is low, the answers provided are still on-topic and relevant to the user's question.
+
+- **Context Recall & Precision**: There are several instances where **context recall** is 0.0, indicating that the context was not helpful in answering the question. However, when context recall is high, **context precision** is often perfect (1.0), showing that when the context is relevant, it is precise and accurate.
+
+- **Answer Correctness**: This metric shows a range of results. Although many answers are correct, a few are only partially correct, suggesting room for improvement in the correctness of generated answers.
+
+The pipeline performs well in generating relevant answers, but some improvements can be made to enhance the faithfulness and correctness of those answers.
+
+The **context recall** metric has room for improvement. There are several cases where relevant context is missing or inadequate, which can impact the overall effectiveness of the pipeline.
+
+## Task 4 - Generate synthetic fine-tuning data and complete fine-tuning of the open-source embedding model
+
+***Deliverable 1*** Swap out your existing embedding model for the new fine-tuned version.  Provide a link to your fine-tuned embedding model on the Hugging Face Hub.
+
+HF Model Link for finetuned model: https://huggingface.co/jeevanions/finetuned_arctic-embedd-l
+
+***Deliverable 2*** How did you choose the embedding model for this application?
+
+The embedding model `snowflake-arctic-embed-l` is ranked 27 in the MTEB ranking which is has embedding dimension of 1024 and 334 Million parameter. Irrespective of its small size, it is a seriously contenting with top players. It is also easier download and train the model with less GPU resource constraint. Low cost but efficient. 
+
+## Task 5 - Assessing Performance
+
+### 1. **Test the fine-tuned embedding model using the RAGAS frameworks to quantify any improvements. Provide results in a table.**
+
+Based on the comparison of the fine-tuned model against the baseline model using the RAGAS framework, here are key metrics evaluated:
+
+| Metric              | Fine-Tuned Model | Baseline Model | Improvement |
+|---------------------|------------------|----------------|-------------|
+| Faithfulness         | 0.5826           | 0.5011         | +8.15%      |
+| Answer Relevancy     | 0.9422           | 0.8765         | +6.57%      |
+| Context Recall       | 0.2716           | 0.2283         | +4.33%      |
+| Context Precision    | 0.4460           | 0.3907         | +5.53%      |
+| Answer Correctness   | 0.6179           | 0.5541         | +6.38%      |
+
+![Comaprison](/ai-safety-chatty/Tasks/Task%205/Task5-ComparisonBaseFineTuned.png)
+![Improvement](/ai-safety-chatty/Tasks/Task%205/Task5-ComparisonBaseFineTunedImprovemant.png)
+
+### 2. **Test the two chunking strategies using the RAGAS frameworks to quantify any improvements. Provide results in a table.**
+
+For chunking strategies (Recursive Character vs. Semantic Chunker), the following metrics were tested:
+
+| Metric              | Recursive Chunking | Semantic Chunking | Improvement |
+|---------------------|--------------------|-------------------|-------------|
+| Faithfulness         | 0.5901             | 0.5826            | -0.75%      |
+| Answer Relevancy     | 0.9500             | 0.9422            | -0.78%      |
+| Context Recall       | 0.3000             | 0.2716            | -2.84%      |
+| Context Precision    | 0.4590             | 0.4460            | -1.30%      |
+| Answer Correctness   | 0.6220             | 0.6179            | -0.41%      |
+
+![Comparison](/ai-safety-chatty/Tasks/Task%205/Task5-SemanticVsRecurstive.png)
+
+> I expected Semantic chunker to perform better and noticed that the retrived document contains less words in the sentences. Also there were some duplicate documents fetched. I would need to spend more time why this has happend. For now with less complexity the RecursiveCharacterText splitter works well and on top implementing advanced RAG techqnique does improve performance.
+
+### 3. **Which one is the best to test with internal stakeholders next week, and why?**
+
+The fine-tuned embedding model paired with **Recursive Character Chunking** is the optimal choice for testing with internal stakeholders. This combination has shown slight improvements in key metrics like **context recall** and **context precision** over semantic chunking. Moreover, the fine-tuned model demonstrates enhanced **faithfulness** and **answer relevancy**, making it more reliable for enterprise-level queries, especially in handling dense documents like the AI Bill of Rights and NIST RMF. While the difference is not dramatic, the Recursive Character Chunking ensures better handling of varied document structures, making it the best candidate for real-world testing.
+
+> For details reports and Notebooks used refer the folder ![Task5 workout](/ai-safety-chatty/Tasks/Task%205/)
+
+## Task 6 - Managing Your Boss and User Expectations
+
+### 1. **What is the story that you will give to the CEO to tell the whole company at the launch next month?**
+
+The story for the CEO should emphasize how the company's investment in AI technology is yielding tangible results and preparing the organization for the future of AI governance. 
+
+**CEO Narrative:**
+
+"Over the past months, our dedicated team of engineers and AI specialists have worked on a groundbreaking Retrieval-Augmented Generation (RAG) application. We’ve tested the system rigorously across 50 internal stakeholders, collecting feedback and refining the performance metrics based on cutting-edge frameworks like RAGAS. This initiative places us at the forefront of ethical AI adoption, ensuring our systems align with upcoming regulatory frameworks such as the 2024 NIST AI Risk Management Framework and the AI Bill of Rights. 
+
+We’ve created a scalable and adaptable AI system that not only answers complex questions related to AI safety but also ensures that our internal teams have the tools to address customer concerns, especially in today's rapidly changing AI landscape. By next month, we’ll be ready to roll out the solution across the enterprise, showing our commitment to innovation, security, and accountability."
+
+
+### 2. **How might you incorporate relevant white-house briefing information into future versions?**
+
+In future versions of the RAG application, it is critical to keep it aligned with national regulations like the executive order on Safe, Secure, and Trustworthy AI. Here's how:
+
+- **Compliance & Transparency:** Ensure that future updates integrate guidelines from the White House’s executive orders on AI governance. This can be achieved by:
+   - **Embedding legal updates** into the knowledge base of the RAG system. This will allow the system to reflect the latest compliance requirements and help internal stakeholders understand how AI aligns with regulations.
+   - Implement an **alert system** within the RAG tool to notify users whenever regulatory changes impact AI practices.
+
+- **Periodic Review:** Set a quarterly update schedule where AI regulations, such as the 270-day update mentioned, are reviewed, and the RAG application is updated accordingly. This ensures that the company stays proactive regarding AI safety and governance.
+
+At the moment these guidelines are loaded into the system offline and in the future we will be rolling out features that keep the system constantly updated on any new regislation. 
+
+This approach will position the company not only as an industry leader but also as a responsible organization that prioritizes safe and ethical AI use.

app.py

@@ -0,0 +1,249 @@
+import os
+from typing import List
+import uuid
+import chainlit as cl
+from chainlit.types import AskFileResponse
+from langchain.memory import ConversationBufferMemory
+from langchain_core.chat_history import BaseChatMessageHistory
+from langchain_community.chat_message_histories import ChatMessageHistory
+from langchain_community.document_loaders import PyMuPDFLoader, TextLoader
+from langchain.prompts import MessagesPlaceholder
+from langchain.prompts import ChatPromptTemplate
+from langchain.chains.history_aware_retriever import create_history_aware_retriever
+from langchain.chains.retrieval import create_retrieval_chain
+from langchain.chains.combine_documents import create_stuff_documents_chain
+from langchain_experimental.text_splitter import SemanticChunker
+from langchain_qdrant import QdrantVectorStore
+from langchain_core.documents import Document
+from qdrant_client import QdrantClient
+from qdrant_client.http.models import Distance, VectorParams
+from langchain_openai import ChatOpenAI
+from langchain_core.runnables.history import RunnableWithMessageHistory
+# from chainlit.input_widget import Select, Switch, Slider
+from dotenv import load_dotenv
+from langchain_huggingface import HuggingFaceEmbeddings
+from langchain.retrievers.contextual_compression import ContextualCompressionRetriever
+from langchain.retrievers.document_compressors import LLMChainExtractor
+
+
+load_dotenv()
+
+BOR_FILE_PATH = "https://www.whitehouse.gov/wp-content/uploads/2022/10/Blueprint-for-an-AI-Bill-of-Rights.pdf"
+NIST_FILE_PATH = "https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf"
+SMALL_DOC = "https://arxiv.org/pdf/1908.10084"  # 11 pages Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
+documents_to_preload = [
+    BOR_FILE_PATH,
+    NIST_FILE_PATH
+    # SMALL_DOC
+]
+collection_name = "ai-safety"
+
+welcome_message = """
+Welcome to the chatbot to clarify all your AI Safety related queries.:
+Now preloading below documents:
+    1. Blueprint for an AI Bill of Rights
+    2. NIST AI Standards
+Please wait for a moment to load the documents.
+"""
+chat_model_name = "gpt-4o"
+embedding_model_name = "Snowflake/snowflake-arctic-embed-l"
+chat_model = ChatOpenAI(model=chat_model_name, temperature=0)
+
+async def connect_to_qdrant():
+    embedding_model = HuggingFaceEmbeddings(model_name=embedding_model_name)
+    qdrant_url = os.environ["QDRANT_URL"]
+    qdrant_api_key = os.environ["QDRANT_API_KEY"]
+    collection_name = os.environ["COLLECTION_NAME"]
+    qdrant_client = QdrantClient(url=qdrant_url,api_key=qdrant_api_key)
+    vector_store = QdrantVectorStore(
+        client=qdrant_client,
+        collection_name=collection_name,
+        embedding=embedding_model,
+    )
+    return vector_store.as_retriever(search_type="similarity_score_threshold",search_kwargs={'k':10,'score_threshold': 0.8})
+
+async def get_contextual_compressed_retriever(retriver):
+
+    base_retriever = retriver
+    compressor_llm = ChatOpenAI(temperature=0, model_name="gpt-4o", max_tokens=4000)
+    compressor = LLMChainExtractor.from_llm(compressor_llm)
+
+    #Combine the retriever with the compressor
+    compression_retriever = ContextualCompressionRetriever(
+        base_compressor=compressor,
+        base_retriever=base_retriever
+    )
+    return compression_retriever
+
+
+def initialize_vectorstore(
+    collection_name: str,
+    embedding_model,
+    dimension,
+    distance_metric: Distance = Distance.COSINE,
+):
+    client = QdrantClient(":memory:")
+    client.create_collection(
+        collection_name=collection_name,
+        vectors_config=VectorParams(size=dimension, distance=distance_metric),
+    )
+
+    vector_store = QdrantVectorStore(
+        client=client,
+        collection_name=collection_name,
+        embedding=embedding_model,
+    )
+    return vector_store
+
+def get_text_splitter(strategy, embedding_model):
+    if strategy == "semantic":
+        return SemanticChunker(
+            embedding_model,
+            buffer_size=3,
+            breakpoint_threshold_type="percentile",
+            breakpoint_threshold_amount=90,
+        )
+
+def process_file(file: AskFileResponse, text_splitter):
+    if file.type == "text/plain":
+        Loader = TextLoader
+    elif file.type == "application/pdf":
+        Loader = PyMuPDFLoader
+
+    loader = Loader(file.path)
+    documents = loader.load()
+    title = documents[0].metadata.get("title")
+    docs = text_splitter.split_documents(documents)
+    for i, doc in enumerate(docs):
+        doc.metadata["source"] = f"source_{i}"
+        doc.metadata["title"] = title
+    return docs
+
+def populate_vectorstore(vector_store, docs: List[Document]):
+    vector_store.add_documents(docs)
+    return vector_store
+
+def create_history_aware_retriever_self(chat_model, retriever):
+    contextualize_q_system_prompt = (
+        "Given a chat history and the latest user question which might reference context in the chat history, "
+        "formulate a standalone question which can be understood without the chat history. Do NOT answer the question, "
+        "just reformulate it if needed and otherwise return it as is."
+    )
+    contextualize_q_prompt = ChatPromptTemplate.from_messages(
+        [
+            ("system", contextualize_q_system_prompt),
+            MessagesPlaceholder("chat_history"),
+            ("human", "{input}"),
+        ]
+    )
+    return create_history_aware_retriever(chat_model, retriever, contextualize_q_prompt)
+
+def create_qa_chain(chat_model):
+    qa_system_prompt = (
+        "You are an helpful assistant named 'Shield' and your task is to answer any questions related to AI Safety for the given context."
+        "Use the following pieces of retrieved context to answer the question."
+        # "If any questions asked outside AI Safety context, just say that you are a specialist in AI Safety and can't answer that."
+        # f"When introducing you, just say that you are an AI assistant powered by embedding model {embedding_model_name} and chat model {chat_model_name} and your knowledge is limited to 'Blueprint for an AI Bill of Rights' and 'NIST AI Standards' documents."
+        "If you don't know the answer, just say that you don't know.\n\n"
+        "{context}"
+    )
+    qa_prompt = ChatPromptTemplate.from_messages(
+        [
+            ("system", qa_system_prompt),
+            MessagesPlaceholder("chat_history"),
+            ("human", "{input}"),
+        ]
+    )
+    return create_stuff_documents_chain(chat_model, qa_prompt)
+
+
+def create_rag_chain(chat_model, retriever):
+    history_aware_retriever = create_history_aware_retriever_self(chat_model, retriever)
+    question_answer_chain = create_qa_chain(chat_model)
+    return create_retrieval_chain(history_aware_retriever, question_answer_chain)
+
+
+def create_session_id():
+    session_id = str(uuid.uuid4())
+    return session_id
+
+
+@cl.on_chat_start
+async def start():
+    msg = cl.Message(content=welcome_message)
+    await msg.send()
+
+    # Create a session id
+    session_id = create_session_id()
+    cl.user_session.set("session_id", session_id)
+
+    retriever = await connect_to_qdrant()
+    contextual_compressed_retriever = await get_contextual_compressed_retriever(retriever)
+
+    rag_chain = create_rag_chain(chat_model, contextual_compressed_retriever)
+
+    store = {}
+
+    def get_session_history(session_id: str) -> BaseChatMessageHistory:
+        if session_id not in store:
+            store[session_id] = ChatMessageHistory()
+        return store[session_id]
+
+    conversational_rag_chain = RunnableWithMessageHistory(
+        rag_chain,
+        get_session_history,
+        input_messages_key="input",
+        history_messages_key="chat_history",
+        output_messages_key="answer",
+    )
+
+    # Let the user know that the system is ready
+    msg.content = msg.content + "\nReady to answer your questions!"
+    await msg.update()
+
+    cl.user_session.set("conversational_rag_chain", conversational_rag_chain)
+
+
+@cl.on_message
+async def main(message: cl.Message):
+    session_id = cl.user_session.get("session_id")
+    conversational_rag_chain = cl.user_session.get("conversational_rag_chain")
+
+    response =  await conversational_rag_chain.ainvoke(
+        {"input": message.content},
+        config={"configurable": {"session_id": session_id},
+                "callbacks":[cl.AsyncLangchainCallbackHandler()]},         
+    )
+    answer = response["answer"]
+
+    source_documents = response["context"]
+    text_elements = []
+    unique_pages = set()
+
+    if source_documents:
+
+        for source_idx, source_doc in enumerate(source_documents):
+            source_name = f"source_{source_idx+1}"
+            page_number = source_doc.metadata['page']
+            #page_number = source_doc.metadata.get('page', "NA")  # NA or any default value
+            page = f"Page {page_number}"
+            text_element_content = source_doc.page_content
+            text_element_content = text_element_content if text_element_content != "" else "No Content"
+            #text_elements.append(cl.Text(content=text_element_content, name=source_name))
+            if page not in unique_pages:
+                unique_pages.add(page)
+                text_elements.append(cl.Text(content=text_element_content, name=page))
+            #text_elements.append(cl.Text(content=text_element_content, name=page))
+        source_names = [text_el.name for text_el in text_elements]
+        
+        if source_names:
+            answer += f"\n\n Sources:{', '.join(source_names)}"
+        else:
+            answer += "\n\n No sources found"
+
+    await cl.Message(content=answer, elements=text_elements).send()
+
+if __name__ == "__main__":
+    from chainlit.cli import run_chainlit
+
+    run_chainlit(__file__)

chainlit.md

@@ -0,0 +1,3 @@
+# Welcome to Chat with Your Text File
+
+With this application, you can chat with an uploaded text file that is smaller than 2MB!

embedding_model.py

@@ -0,0 +1,58 @@
+
+import tiktoken
+import os
+from langchain_openai import OpenAIEmbeddings
+from langchain_community.embeddings import HuggingFaceBgeEmbeddings
+import torch
+from transformers import AutoModel, AutoTokenizer
+from transformers import AutoModel, AutoTokenizer
+from langchain_huggingface import HuggingFaceEmbeddings
+
+# def get_embeddings_model_bge_base_en_v1_5():
+#     model_name = "BAAI/bge-base-en-v1.5"
+#     model_kwargs = {'device': 'cpu'}
+#     encode_kwargs = {'normalize_embeddings': False}
+#     embedding_model = HuggingFaceBgeEmbeddings(
+#         model_name=model_name,
+#         model_kwargs=model_kwargs,
+#         encode_kwargs=encode_kwargs
+#     )
+#     return embedding_model
+
+# def get_embeddings_model_bge_en_icl():
+#     model_name = "BAAI/bge-en-icl"
+#     model_kwargs = {'device': 'cpu'}
+#     encode_kwargs = {'normalize_embeddings': False}
+#     embedding_model = HuggingFaceBgeEmbeddings(
+#         model_name=model_name,
+#         model_kwargs=model_kwargs,
+#         encode_kwargs=encode_kwargs
+#     )
+#     return embedding_model , 4096
+
+# def get_embeddings_model_bge_large_en():
+#     model_name = "BAAI/bge-large-en"
+#     model_kwargs = {'device': 'cpu'}
+#     encode_kwargs = {'normalize_embeddings': False}
+#     embedding_model = HuggingFaceBgeEmbeddings(
+#         model_name=model_name,
+#         model_kwargs=model_kwargs,
+#         encode_kwargs=encode_kwargs
+#     )
+#     return embedding_model
+
+def get_embeddings_openai_text_3_large():
+    embedding_model = OpenAIEmbeddings(model="text-embedding-3-large")
+    dimension = 3072
+    return embedding_model,dimension
+
+# def get_embeddings_snowflake_arctic_embed_l():
+#     current_dir = os.path.dirname(os.path.realpath(__file__))
+#     model_name = "Snowflake/snowflake-arctic-embed-l"
+#     tokenizer = AutoTokenizer.from_pretrained(f"{current_dir}/cache/tokenizer/{model_name}")
+#     model = AutoModel.from_pretrained(f"{current_dir}/cache/model/{model_name}")
+#     return model,1024
+
+def get_embeddings_snowflake_arctic_embed_l():
+    embedding_model = HuggingFaceEmbeddings(model_name="Snowflake/snowflake-arctic-embed-l")
+    return embedding_model,1024