Can Autonomous AI Systems Be Governed Like Software? 

Key takeaways

• Software governance is the foundation, but autonomous AI also needs runtime controls for authority, tools, and actions
• Agentic AI governance must define what an agent can access and do
• Identity, permissions, monitoring, human oversight, and auditability must remain active throughout the agent’s lifecycle
• People and organizations remain accountable for consequential outcomes
• Low-risk actions can be automated, while high-risk actions require stronger controls and human oversight

Autonomous AI is moving enterprise AI from generating answers to taking actions. Agents can plan tasks, access data, invoke tools, and execute workflows, creating a new governance challenge. Deloitte Insights reveals that while 74% of enterprises plan rapid expansion of agentic AI (autonomous systems that take independent actions), only 21% maintain structured governance aligned to their autonomy level. This creates a significant execution gap (the gap between designed controls and runtime reality), particularly when unexpected behavior changes during execution, known as runtime drift.

So, can autonomous AI systems be governed like software? Yes, but traditional software governance addresses code and deployment; autonomous AI requires additional runtime controls. Enterprises need software engineering discipline combined with controls for identity, authority, tool access, runtime behavior, and accountability. The shift is already reflected in global guidance. Singapore’s Model AI Governance Framework for Agentic AI emphasizes bounded autonomy, controlled access to tools and data, meaningful human accountability, and lifecycle governance.

What Is AI Governance for Autonomous AI Systems?

Effective AI governance integrates policies, processes, technical controls, ownership structures, and oversight mechanisms that determine how artificial intelligence is designed, deployed, operated, monitored, and retired. For autonomous AI systems, the governance boundary expands beyond the model to the complete agentic AI system.

That system may include:

A practical AI governance framework should therefore answer six fundamental questions:

1. What is the agent authorized to accomplish?
2. Which data and systems can it access?
3. Which tools can it invoke?
4. Which actions can it take independently?
5. Which actions require human approval?
6. Who owns the agent and remains accountable for its outcomes?

To answer these questions in practice, enterprises need six governance pillars that translate these requirements into implementable controls. 

NIST’s AI Risk Management Framework provides a strong foundation through its Govern, Map, Measure, and Manage functions. For autonomous systems, these principles need to extend into runtime operations, where decisions and actions occur. This shift from governing how AI is built to governing how it acts is what makes autonomous AI governance different from traditional software governance.

How Does AI Governance Work Differently for Autonomous Systems?

Traditional software governance focuses on code, releases, testing, access controls, versioning, and change management. These controls remain essential for autonomous AI, but they do not fully address how an agent acts at runtime.

An autonomous agent can interpret a goal, choose a tool, access data, evaluate results, and decide its next action. This creates the execution authority gap: traditional software executes predefined logic, while an AI agent operates within delegated authority.

Closing the execution authority gap

Consider the difference:

• Traditional control: “This application can access the customer database.”
• Agentic control: “This agent can read specific customer data for an approved task but cannot modify or export it.”

Agentic AI governance connects purpose, identity, permissions, tools, and actions. Because prompts, context, memory, retrieved data, tool results, orchestration, and model updates can influence execution, governance must monitor the complete runtime environment, not just the model.

Why Is Agentic AI Governance Important for Enterprise Scaling?

While autonomy amplifies enterprise productivity, it also increases operational exposure. Low-impact tasks carry minimal risk, but systems that deploy code, approve refunds, or modify records need proportional oversight. According to Gartner Research on AI TRiSM (Trust, Risk, and Security Management), dedicated runtime controls help organizations mitigate data leakage, hallucinations, and unauthorized agentic drift. 

The design principle is: grant autonomy proportional to risk and business impact. 
 

6 Critical Pillars Every AI Governance Framework Must Control

A mature AI governance framework must manage six core pillars to ensure autonomous AI systems operate safely and securely:

1. Identity and Authority: Assign every autonomous agent a distinct machine identity, defined purpose, named owner, and strict authorization scope rather than sharing generic credentials.
2. Tools and Data: Enforce least-privilege access across all business systems and govern which Model Context Protocol servers an agent can discover or invoke.
3. Runtime Behavior: Track real-time telemetry like abnormal action volumes, unusual tool calls, and boundary-crossing attempts while the system operates.
4. Human Oversight: Apply risk-based human-in-the-loop controls, reserving manual checkpoints for high-impact decisions rather than slowing down low-risk tasks.
5. Auditability: Maintain immutable audit trails linking agent identity, authorization, tool usage, policy checks, and outcomes for compliance and AI accountability.

6. Multi-Agent Delegation: Establish clear rules for authority transfer, inherited restrictions, and final accountability when interconnected agents collaborate on complex workflows.
    

   

Can AI Governance Govern Autonomous AI Like Software?

No. Software governance is the foundation, not the complete solution. Traditional software controls remain indispensable, including:

• Secure development
• Version control
• Testing
• Change management
• Identity and access management
• Observability
• Incident response

But autonomous AI adds another governance layer:

• Model and behavior evaluation
• Autonomy boundaries
• Tool authorization
• Runtime policy enforcement
• Human intervention
• Agent identity
• Behavioral telemetry
• AI accountability

Software governance controls how technology is developed and operated. AI governance also controls how autonomous AI systems behave, make decisions, use delegated authority, and interact with their environment. Effective autonomous AI governance therefore needs both engineering and operational controls. 

How Can Enterprises Implement an AI Governance Framework?

Enterprises moving autonomous AI into production can start with six practical steps.

• Inventory the AI Estate: Identify agents, models, tools, data sources, workflows, vendors, and environments.
• Define Delegated Authority: Specify exactly what each agent can access, change, approve, and initiate.
• Classify Risk: Consider business impact, data sensitivity, financial exposure, regulatory consequences, and reversibility.
• Enforce Controls Technically: Use identity controls, permissions, policy engines, approval gates, and tool restrictions rather than relying on instructions alone.
• Monitor Continuously: Combine security, compliance, operational, and behavioral telemetry to detect abnormal activity and policy violations.
• Maintain Evidence and Accountability: Connect every consequential action to an owner, authorization, control decision, and outcome. 

The objective is not to make agents behave like deterministic software. The objective is to make their operating boundaries deterministic even when their reasoning is not.

What Does Responsible AI Look Like for Autonomous Systems?

Responsible AI becomes meaningful when principles are translated into enforceable engineering controls.

• Fairness requires measurable evaluation.
• Privacy requires clear data boundaries.
• Security requires controlled identities and permissions.
• Transparency requires usable evidence and traceability.
• Human oversight requires actual intervention and shutdown mechanisms.

Most importantly, accountability remains human even when execution is autonomous. An autonomous agent may execute a workflow independently, but that does not transfer organizational accountability to the agent itself. Enterprises still need clearly defined owners responsible for how the system is designed, deployed, monitored, and used.

Keynote

The goal is not maximum autonomy or maximum human oversight. It is controlled autonomy: greater freedom for low-risk actions, stronger controls as risk rises, and human intervention where consequences are significant. Software governance controls the technology. AI governance controls the risk. Agentic AI governance controls delegated authority. At TO THE NEW, we help enterprises embed engineering, security, observability, and governance into AI and cloud architectures, enabling organizations to scale autonomous AI with control, accountability, and trust.