A Framework for Automated Measurement of Responsible AI Harms in Generative AI Applications
akhaliq
Abstract
We present a framework for the automated measurement of responsible AI (RAI) metrics for large language models (LLMs) and associated products and services. Our framework for automatically measuring harms from LLMs builds on existing technical and sociotechnical expertise and leverages the capabilities of state-of-the-art LLMs, such as GPT-4. We use this framework to run through several case studies investigating how different LLMs may violate a range of RAI-related principles. The framework may be employed alongside domain-specific sociotechnical expertise to create measurements for new harm areas in the future. By implementing this framework, we aim to enable more advanced harm measurement efforts and further the responsible use of LLMs.
Community
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- GPTFUZZER : Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023)
- Design-Inclusive Language Models for Responsible Information Access (2023)
- Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation (2023)
- ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models (2023)
- Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs (2023)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper