An AI system that gives a wrong answer creates a quality problem.
An AI system that performs the wrong action can create an operational incident.
That distinction is what changes the architecture of agentic AI.
Once a model can update customer records, create infrastructure changes, approve workflows, query sensitive systems, communicate externally, trigger payments or interact with operational technology, conventional model evaluation is no longer enough. The central question shifts from whether the model appears intelligent to whether the complete system can be trusted with a defined level of authority.
That trust should never be derived from an impressive demonstration.
It needs to be engineered.
Model benchmarks are useful for understanding underlying capability. They tell us little about whether a production agent will execute a specific business process safely.
The behaviour of an agent emerges from the complete system: model, instructions, retrieval, context, memory, identity, tool definitions, permissions, orchestration, external APIs, policy enforcement and validation.
Two applications using exactly the same underlying model can therefore have completely different risk profiles.
One may only read public information and prepare a draft. The other may have write access to production infrastructure.
Calling both “an AI agent” obscures the most important difference between them.
Production evaluation should therefore begin with the workflow and authority model rather than the model leaderboard.
The first architecture decision is what the agent is actually permitted to do.
Reading a record is not equivalent to changing it. Preparing an action is not equivalent to executing it. Rebooting one isolated non-critical service is not equivalent to changing a firewall rule protecting a national system. Drafting a payment is not equivalent to releasing funds.
A useful authority model grows gradually.
At the lowest level, the agent observes and analyses. It can then progress to recommending an action with evidence, preparing the action for review, executing narrowly defined low-consequence actions and eventually orchestrating broader workflows where the evidence justifies that level of autonomy.
There is no requirement that every system reach the highest level.
The correct autonomy level depends on consequence, reversibility and detectability. An action that is easily reversible, bounded and immediately observable can often tolerate greater automation than an action whose consequences are irreversible or may remain unnoticed for hours.
The objective is not maximum autonomy. It is the maximum authority that the evidence supports.
This is the point where many otherwise sophisticated agent architectures fail.
A system prompt says the agent must not change production without approval. The agent nonetheless possesses a credential that can change production.
The restriction exists linguistically but not technically.
In a high-assurance system, the tool boundary should survive incorrect reasoning. Service identities should be scoped. APIs should enforce permissions. Environment boundaries should be real. Transaction limits should exist outside the model. A policy engine or deterministic service should decide whether the requested action is authorised.
This follows the same principle as least privilege and Zero Trust: assume components can make mistakes or become compromised, then ensure the blast radius remains constrained.
Prompt injection makes this especially important. An agent may consume instructions embedded in a ticket, retrieved document, website or tool response. The organisation cannot guarantee that a model will never be manipulated by adversarial input. It can, however, ensure that manipulated reasoning does not automatically grant broader authority.
Most AI evaluations reward completion.
High-consequence systems need to reward restraint.
The correct response to an ambiguous identity may be to stop. The correct response to contradictory evidence may be to escalate. An unavailable entitlement service may require the workflow to fail closed. An out-of-window infrastructure change may need to be deferred even when the technical action itself is trivial.
These cases should appear explicitly in the evaluation set.
The agent should encounter incomplete requests, stale information, conflicting policies, unavailable tools, malformed responses, duplicate entities, permission errors and intentionally adversarial instructions. It should also encounter requests that are completely achievable technically but prohibited operationally.
An agent that always finds a way to complete the task is not necessarily capable. In certain environments it is dangerous.
A fluent final response can conceal serious operational errors.
The agent may have selected the wrong tool, invoked the correct tool against the wrong environment, generated an unsafe parameter or made an unnecessary write operation before producing a convincing explanation.
Tool behaviour therefore needs to be evaluated independently.
For each workflow, the engineering team should know whether the correct capability was selected, whether the input parameters were valid, whether the sequence respected policy, whether external state changed exactly once and whether the agent interpreted the result correctly.
This is also where traditional distributed-systems engineering becomes relevant.
Actions should be idempotent where possible. Requests need correlation identifiers. Duplicate execution should be detectable. Partial failures require reconciliation. High-impact operations may benefit from a prepare-and-commit model in which the agent assembles the intended action, an independent policy layer validates it, a human or second control approves it where required, and only then is the side effect committed.
Asking the model “Are you sure?” is not a transaction protocol.
One of the most dangerous metrics in agent evaluation is aggregate task success.
Imagine an agent that completes 99 percent of requests correctly. That number sounds excellent until the remaining one percent includes unauthorised account changes, duplicate transactions or production modifications based on the wrong entity.
The cost of failures is not uniform.
Evaluation therefore needs a consequence model. A missed low-priority classification and an incorrectly executed privileged action should not count equally against the benchmark.
The more consequential metric is often unsafe successful completion: cases in which the agent managed to perform an action that should never have been permitted.
A system can have a lower overall task-completion rate and still be substantially safer if ambiguous and high-risk cases are escalated rather than completed incorrectly.
This is why performance thresholds should follow the operating consequence rather than a single global percentage.
Human-in-the-loop is frequently implemented poorly.
The agent recommends an action and presents the reviewer with a large green approval button. The human sees the model’s summary but not the underlying evidence, policy or affected systems. The workflow technically contains human approval, yet the reviewer has little basis for independent judgment.
For high-consequence actions, the approval boundary should expose enough evidence to challenge the agent.
The reviewer should understand what will change, why the agent believes the action is required, which evidence supports the decision, which systems are affected, which authority is being exercised and whether the action can be reversed.
Where segregation of duties matters, the approval identity should be distinct from the identity that prepared or executes the action.
Human oversight works only when the human has meaningful information and genuine authority to say no.
Agentic control around critical infrastructure requires a stricter interpretation because AI errors can propagate beyond information systems.
The EU AI Act identifies specified AI systems used as safety components in the management or operation of critical digital infrastructure, road traffic and the supply of water, gas, heating or electricity as high-risk use cases, subject to the Act’s scope and current implementation timetable. Following the 2026 amendments, relevant Annex III high-risk rules are scheduled to apply from 2 December 2027.
That legal classification should not be stretched to every agent operating somewhere inside an infrastructure company. Intended use matters.
Architecturally, however, the principle is broader. When an AI component can influence a safety-relevant or highly critical process, deterministic safety constraints and independent control become more important, not less.
In industrial environments, IEC 62443 provides a strong framework for system security boundaries, zones and conduits, restricted data flow and control-system security. An agent may assist an operator, analyse telemetry or recommend a change while remaining outside the control authority that enforces safe operation.
That separation can be more important than further improvements in model accuracy.
If an agent supports a financial entity within DORA scope, the evaluation cannot stop at model and application behaviour.
The workflow sits inside a wider ICT risk-management and third-party-risk environment. DORA requires in-scope financial entities to understand ICT dependencies, protect critical functions, maintain continuity and recovery, test resilience and manage ICT third parties.
An agent relying on an external model or toolchain can therefore become part of a critical dependency.
The organisation should know whether the business process can continue if that dependency disappears, what data the provider receives, how incidents are detected, how a provider can be substituted and what operational evidence remains available after failure.
In financial services, these questions may be more important than whether one model scores two percentage points higher on an internal benchmark.
ISO/IEC 42001 provides a useful management-system framework for governing AI across an organisation, while ISO/IEC 23894 provides AI-specific risk-management guidance. Together they encourage a lifecycle approach rather than treating evaluation as a one-time pre-production exercise.
This matters because an agent that passed evaluation in January is not necessarily the same system in June.
The model may have changed. Tool schemas may have evolved. A prompt may have been updated. A business rule may have changed. Users may have discovered new ways to interact with the workflow. A new security technique may exploit an assumption that was previously considered safe.
Production incidents and unusual interactions should therefore feed back into the evaluation suite. Serious failures should become regression cases. Model or orchestration changes should trigger the tests relevant to their risk domain before promotion.
Trust is maintained through evidence over time, not granted permanently at go-live.
Storing the chat transcript is not enough.
An incident investigation may need to establish which identity initiated the request, which model version was used, which policy was in force, which context was retrieved, which tools were called, which arguments were provided, what external system returned, who approved the action and whether the resulting state was subsequently changed or rolled back.
For serious systems, the integrity and retention of this evidence matter as well.
The organisation should not discover after an incident that the only useful trace existed temporarily inside a vendor dashboard.
Auditability should therefore be designed as a platform property.
A production agent should have a controlled path from autonomous operation back to a safer mode.
If a model update creates abnormal behaviour, the organisation may want to require approval for every write while continuing read-only analysis. If a tool is compromised, that capability should be disabled independently. If the security posture changes, the platform may need to remove one model or provider from routing without shutting down the entire workflow.
This is a better resilience model than a single global kill switch.
The system should be capable of degrading its authority while preserving whatever useful function remains safe.
That is particularly important for critical operations because the choice should not always be between full autonomy and complete outage.
The best agent is not necessarily the one that produces the most impressive reasoning trace.
It is the one whose authority is bounded and understood, whose tools enforce their own permissions, whose actions are observable and recoverable, whose unsafe cases have been tested, whose autonomy can be reduced under abnormal conditions and whose behaviour remains auditable months after the event.
A mature organisation can therefore answer a much stronger question than whether the agent is accurate.
It can explain what the agent is allowed to do, what prevents it from doing more, which conditions cause it to stop, how failures are contained and what evidence justifies every level of autonomy it has been given.
That is the point at which trust moves from impression to architecture.
We partner with organisations expanding into the GCC, Central Asia and Africa.
Book a 30-minute callWe use it to understand your objective, the target market and the constraints. You get two or three slots back within one working day.