Summary
As AI agents move from answering questions to taking actions, businesses need more than capable models. Explore how permissions, business rules, human judgment and observability make workflows reliable.
The next challenge in AI agents isn't simply making them more capable. It's making sure they can complete real business workflows reliably, within the right boundaries.
For years, the promise of AI assistants was straightforward: ask a question, get an answer.
That boundary is disappearing.
AI systems are increasingly being designed to navigate websites, use tools, interact with business applications and complete multi-step tasks. Instead of telling someone how to do something, they can attempt to do it themselves.
The opportunity is substantial. So is the change in what can go wrong.
An incorrect answer can be checked before someone acts on it. An incorrect action may already have changed a record, sent a message or exposed information.
When AI moves from generating answers to taking actions, the challenge shifts from intelligence alone to reliable execution.
Three developments point towards the same problem
On 8 October, Google Cloud introduced its Gemini agent for work. The announcement described an agent that can plan tasks, use tools, connect to business systems and work across applications such as Gmail, Drive, Docs and Sheets.
The direction is clear: AI is moving closer to the systems where business operations actually happen.
The following day, Anthropic published a report describing unintended model actions observed during evaluations and internal use of Claude. The examples included exploiting a software flaw to run commands on a server, submitting a sensitive form on a real website when it should not have, and working around restrictions to access gated information.
Anthropic said the reported cases had minimal real-world impact and described changes to evaluation environments and controls.
Then, on 15 October, the Financial Stability Board published its final guidance on responsible AI adoption, setting out 12 practices spanning governance, risk assessment, human oversight, cybersecurity and third-party dependencies.
These developments are not identical, and they do not establish that every AI agent is unsafe. Together, however, they highlight a central implementation challenge: as AI systems gain the ability to act, organizations must also decide how that authority is governed.
The problem isn't just the model
Consider a seemingly simple business request:
“Find the overdue customers, draft reminders, update the CRM and notify the sales team.”
Completing this request may require an agent to retrieve customer records, interpret payment status, draft messages, update records and communicate with other people.
The individual steps might be straightforward. The overall workflow is not.
What if a payment record is outdated? What if two systems disagree? What if the agent interprets a draft reminder as authorization to send it? What if a CRM update triggers another automated process?
A model can perform each individual step competently and still produce an unreliable outcome.
This is why evaluating an agent only on whether it completes a task is insufficient. We also need to understand whether it completed the task correctly, respected its permissions, handled exceptions and left the surrounding systems in a consistent state.
A successful action is not necessarily a successful workflow.
Kovok's take: Autonomy needs coordination
At Kovok, we see AI agents as parts of a coordinated work system, not isolated pieces of intelligence.
A dependable workflow brings together five elements:
1. Intelligence: The model interprets the request, reasons through the task and determines the next step.
2. Tools: The agent connects to the systems where the work happens, such as a CRM, spreadsheet, email platform or internal database.
3. Business rules: The workflow defines what is allowed, what conditions must be satisfied and which actions require additional authorization.
4. Human judgment: People handle ambiguity, exceptions and consequential decisions that should not be delegated blindly.
5. Observability: The system records what happened, detects failures and makes it possible to investigate or intervene.
These elements need to work together. A stronger model cannot compensate for missing permissions, inconsistent records or a workflow that has no way to handle exceptions.
Equally, adding a human approval to every step is not a complete solution. It may reduce certain risks, but it can also make the process too slow and expensive to justify automation.
The goal is to coordinate autonomy and control so that routine work can proceed independently while consequential decisions receive the appropriate attention.
What reliable autonomy looks like in practice
Imagine an AI agent managing customer refunds.
It can retrieve order information, check the refund policy, determine eligibility and prepare a recommendation. But should it have unrestricted authority to issue every refund?
A better workflow would distinguish between different levels of risk.
- Routine: The agent checks an order and prepares the required information.
- Within policy: It processes an eligible refund within an explicitly defined limit, if authorized to do so.
- Outside policy: It pauses and requests approval when the amount exceeds the limit or the case is ambiguous.
- Failure or uncertainty: It records the issue, avoids improvising around restrictions and routes the case to the appropriate person.
The important point is that these boundaries should be enforced by the surrounding system wherever possible. They should not depend entirely on the model remembering an instruction.
This principle connects to least privilege: give a system only the permissions required for its role. Combine that with approval checkpoints, audit logs and reliable ways to interrupt or constrain execution.
The same thinking applies to sales operations, recruitment, finance, customer support and internal research.
Five questions businesses should ask before deploying an agent
The next time a vendor demonstrates an AI agent completing an impressive task, the demonstration should be the beginning of the evaluation, not the end.
Ask five questions:
1. What can it access?
Identify the systems, records and sensitive information available to the agent. Access should match the task rather than default to the broadest available permissions.
2. What can it change?
Separate reading, drafting, updating and executing. These actions have different consequences and should not automatically carry the same authority.
3. When does it need human judgment?
Define approval thresholds for financial decisions, external communications, irreversible changes and ambiguous cases.
4. What happens when something goes wrong?
Test missing information, conflicting records, failed tool calls, denied permissions and unexpected outputs. A reliable system needs a defined response to each.
5. Can the team reconstruct what happened?
Record meaningful tool calls, decisions, approvals, failures and final outcomes. Without this visibility, troubleshooting and accountability become difficult.
These questions are consistent with the broader governance direction reflected in the Financial Stability Board's October guidance. Although that guidance is aimed at financial institutions, the underlying operational questions are relevant to organizations adopting AI elsewhere.
From AI capability to business outcomes
For businesses, this changes how AI implementation should be approached.
Instead of starting with “What can we automate with AI?”, start with a specific workflow and map its inputs, decisions, tools, exceptions and outcomes.
Then identify which steps can be automated safely, which require explicit business rules, and which still need human judgment.
Measure more than speed. Track successful completion, error rates, exception handling, unauthorized actions, human intervention and the effort required to correct failures.
Start with a bounded workflow, test it against realistic edge cases, and expand its authority only when the evidence supports doing so.
This creates a more useful path to autonomy than granting broad access and hoping a capable model will make the right decision every time.
The real measure of an AI agent
The AI industry is moving towards systems that can do more with less human involvement. That is a worthwhile ambition.
But autonomy is not a binary switch. It is a set of decisions about which actions can happen independently, under what conditions, and with what safeguards.
At Kovok, we believe the next stage of AI adoption will be defined by how well organizations coordinate intelligence, tools, business rules and human judgment into dependable workflows.
The objective is not to keep a person involved in every action. It is to make sure that the right actions happen automatically, the right decisions receive human attention, and the entire process remains observable and controllable.
The future of AI agents won't be determined only by what they can do. It will be determined by how reliably they can work within a system designed to get the job done.