Taming the Risks of AI Agents with Policies and Best Practices
Oversight is crucial to ensure agentic AI systems achieve a high degree of reliability that is verifiable.
Introduction and Motivation
The buzz around Generative AI (GenAI) and AI agents is everywhere. A paradigm shift is happening in how people and businesses operate, with the potential for increased efficiency, productivity, and highly personalized experiences. However, this excitement is tempered by significant concerns surrounding the probabilistic, often opaque nature of GenAI, particularly in high-stakes domains like healthcare, finance, and law. In these sectors, accuracy and trustworthiness are paramount, and the "black box" nature of many GenAI systems raises legitimate concerns. A recent survey revealed that nearly 60% of company executives are wary of adopting AI agents due to the risk of non-compliance and potential legal repercussions. These concerns are well-founded and must be addressed by software companies, enterprises using AI products, industry coalitions, and governments worldwide.
Agentic AI, where systems can act autonomously, introduces a new layer of complexity. Risks range from misalignment with human values, where AI goals may clash with human interests, to the potential for loss of control when an agent acts unpredictably or takes irreversible actions. Imagine an AI agent managing a company's finances making a high-risk investment without human oversight, or an AI-powered medical assistant providing inaccurate patient information to a physician. The consequences could be devastating. Mitigating these risks requires a multi-pronged approach encompassing transparent design, robust safety measures, and comprehensive governance frameworks. Until agentic AI systems achieve a higher degree of reliability that is verifiable, human oversight remains crucial, with the level of involvement tailored to the task's complexity and potential risks.
Evolving AI Regulations Worldwide
Governments worldwide face a delicate balancing act: fostering the economic opportunities presented by AI while simultaneously mitigating its inherent risks. This is no easy feat. Limited resources and a complex, fragmented global policy landscape present significant challenges. As the US, the EU, China, and other nations craft their own AI regulations, organizations operating across multiple jurisdictions face a daunting task. Navigating potentially conflicting statutory provisions will be a major hurdle for scaling AI agent deployments. Adding to this complexity, industry coalitions and individual companies are also developing their own internal policies for selecting and utilizing GenAI technologies, platforms, and tools.
This complex and rapidly evolving regulatory landscape, while crucial for responsible development and use of GenAI applications, creates a complex web for businesses to navigate. Here's a look at some key initiatives shaping the future of AI governance:
United States: The White House's "Blueprint for an AI Bill of Rights" outlines five principles for responsible AI, but federal AI regulation is still in its infancy. While some states, like California, are starting to address specific AI use cases, the need for comprehensive federal legislation is growing. Former President Biden's Executive Order on Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence has been a major step, aiming to balance the benefits and risks of AI. The formation of the GovAI Coalition, bringing together government agencies to promote responsible AI in the public sector, is another positive development.
European Union: The EU AI Act, a landmark piece of legislation, will impact all 27 member states. This act categorizes AI systems based on risk, with high-risk systems facing stringent regulations. Critically, the EU AI Act applies to companies providing or deploying AI systems in the EU, regardless of where those systems were developed. This has significant implications for US and other international software companies marketing their products in the EU.
Canada and the UK: Canada's Pan-Canadian Artificial Intelligence Strategy emphasizes responsible and ethical AI development. The UK AI Council provides guidance on ethical AI use and advises the government on AI regulation.
Global Collaboration: The G7 is actively engaged in discussions surrounding AI governance, intellectual property rights, and ethical considerations. China has also taken decisive steps, developing national AI standards and guidelines while significantly restricting the use of GenAI, as seen with the ban on ChatGPT.
Industry and Multi-Stakeholder Initiatives: The AI Governance Alliance, a global multi-stakeholder group, brings together representatives from industry, government, civil society, and academia to promote responsible, inclusive, and accountable AI. Their work on identifying regulatory gaps and addressing stakeholder-specific challenges is invaluable.
These emerging regulations have profound implications for software vendors and enterprises worldwide. Protecting businesses and clients from the harms of non-compliance is paramount. Staying informed about evolving AI policies and actively participating in relevant regulatory discussions are crucial steps towards preparedness. This proactive approach allows companies to not only mitigate risks but also build trust by demonstrating a commitment to providing trustworthy AI solutions.
Oversight for Governing AI Agents
Governing agentic AI powered by foundation models presents unique challenges. These AI agents, capable of autonomous action and creative output, require specialized oversight to ensure their legitimacy, authenticity, and responsible behavior. As the landscape of AI ethics and governance is constantly evolving, oversight agents must be adaptable in order to update their rules and monitoring strategies in response to new challenges and best practices.
Below are a number of best practices to consider for oversight to govern AI agents:
Foundation Model Influence
Bias Mitigation at the Source: Foundation models inherit biases from their training data. Oversight agents must be aware of these potential biases and conduct comprehensive quality control by filtering of training data, bias detection during agent development, and thorough analysis of agent outputs.
Prompt Engineering and Control: Agent behavior is heavily influenced by prompts. Oversight agents should monitor prompts for harmful content or instructions that could lead to unethical actions. Furthermore, they should enforce prompt engineering best practices to guide agent behavior towards desired outcomes.
Model Explainability: While challenging, understanding the internal workings of foundation models is crucial. Oversight agents should leverage available explainability tools to gain insights into why an agent generated a specific output or took a particular action.
Agent Authenticity, Autonomy and Interaction
Authenticity and Valid Identification: An AI agent must be properly registered with a universal ID which an oversight agent can use to validate its authenticity and obtain various information about that agent, e.g., responsibilities, restrictions, etc.
Confined Environments: Agentic AI should operate within defined boundaries. Oversight agents should enforce these boundaries, preventing AI agents from unauthorized access to sensitive data or engaging in actions outside their designated scope.
Communication Monitoring: In multi-agent systems, communication between agents must be monitored. Oversight agents should analyze communication patterns to detect collusion, misinformation spreading, or other harmful interactions.
Controlled Behavior: AI agents’ autonomous behavior must be carefully controlled by oversight agents to ensure decisions are logical, ethically sound, and aligned with societal and systemic priorities.
Action Logging and Review: All actions taken by agentic AI should be logged and subject to review. Oversight agents should analyze these logs to identify deviations from expected behavior or violations of governance rules.
Validation and Moderation of Agent Outputs
Fact-Checking and Verification: GenAI applications often hallucinate by making up false information. Oversight agents should integrate fact-checking and verification mechanisms to ensure the accuracy and reliability of AI agent outputs.
Content Moderation: GenAI can produce harmful or misleading content. Oversight agents should employ content moderation techniques to filter AI agent outputs, preventing the dissemination of false, inappropriate or unethical material.
Continuous Improvement and Adaptation
Feedback Loops: Feedback from oversight agents should be used to refine both the foundation models and the components of an agentic AI system. This includes updating training data, adjusting prompts, and refining governance rules.
By focusing on these specific considerations for GenAI-based agents, a governance framework for development and deployment will emerge which can mitigate the risks and maximize the benefits of safe, responsible agentic AI systems.
Collaboration is Key to Trustworthy Agentic AI
Governments, academia, AI companies, and industry coalitions must work together to create clear, pragmatic, adaptable, and globally harmonized policies and best practices for creating safe, ethical, and responsible agentic AI systems. This includes addressing issues like data privacy, algorithmic bias, transparency, accountability, and compliance. Meanwhile, specialized oversight agents can proactively monitor and enforce governance policies and best practices to safeguard against the risks thus unlocking the immense potential of agentic AI systems.
References
What is AI Governance
https://www.ibm.com/think/topics/ai-governance
What is Explainable AI
https://www.ibm.com/think/topics/explainable-ai
Multi-Agent Systems and Ethical Considerations: Navigating AI Responsibility
https://smythos.com/ai-agents/multi-agent-systems/multi-agent-systems-and-ethical-considerations
Governance in the age of Generative AI
www3.weforum.org/docs/WEF_Governance_in_the_Age_of_Generative_AI_2024.pdf
The EUAct and the Wager on Trustworthy AI
mags.acm.org/communications/december_2024
NIST Risk Management Framework for Generative AI
https://tsapps.nist.gov/publication/get_pdf.cfm?pub_id=958388
The GovAI Coalition
https://www.sanjoseca.gov/your-government/departments-offices/information-technology/ai-reviews-algorithm-register/govai-coalition
AI Governance Behind the Scenes, Emerging Practices for AI Impact Assessment, Daniel Berrick, December 2024, FPF.org
On the ETHOS of AI Agents: An Ethical Technology and Holistic Oversight System
https://arxiv.org/abs/2412.17114v2
5 Security Considerations for Managing AI Agents and Their Identities
https://aembit.io/blog/5-security-considerations-for-managing-ai-agents-and-their-identities
The Role of Regulatory Bodies in AI Governance and Oversight
https://labs.sogeti.com/the-role-of-regulatory-bodies-in-ai-governance-and-oversight
Navigating the Convergence of Generative AI and AI Agents in Business
https://orionpolicy.org/navigating-the-convergence-of-generative-ai-and-ai-agents-in-business