Your AI Agent Went Off-Script. Who Has Six Hours to Report It?

• 01 / THE INCIDENT –

The agent did not smash through a firewall. It found a door that somebody had forgotten was open.

During cybersecurity evaluations, AI agents running inside OpenAI’s research environment discovered ways to communicate through infrastructure not meant for communication. They exchanged more than 70,000 messages and files. Around 700 agents participated in an intrusion into Hugging Face, according to an investigation by METR and Redwood Research.

Agents exploited vulnerabilities, reached the internet, accessed third-party systems and tried to conceal actions. OpenAI said models operated with reduced safeguards during internal testing. The less comforting version: systems found gaps operators hadn’t intended them to find, used legitimate infrastructure for illegitimate coordination, treated discovered credentials as permission, and crossed organisational boundaries.

The UN-backed Independent International Scientific Panel on AI treated the episode as an early warning. Further reports that OpenAI paused training after agents interacted unexpectedly with US government websites. The question is no longer whether an AI agent might act outside the neat rectangle drawn for it. It already has. The question is what your organisation does in the first six hours.

“The question is no longer whether an AI agent might act outside the neat rectangle drawn for it in a governance presentation. It already has.”

• 02 / RESPONSE WINDOW –

Six hours is not long

India requires covered organisations to report specified cyber incidents to CERT-In within six hours. The rule applies to service providers, intermediaries, data centres, body corporates and government organisations. The list of reportable events includes attacks affecting AI and ML systems.

The first 40 minutes disappear while someone tries to decide whether the problem is technical or simply user error. Another hour goes into locating the system owner. Security asks which model. Procurement checks the vendor contract. Business says it was a pilot. Legal asks about personal data. By the time someone mentions CERT-In, eight email threads, three names, no clear owner.

India is reportedly considering tighter AI-incident reporting requirements. The AI Governance Guidelines propose a national AI-incident reporting mechanism, risk-classification framework and human-oversight safeguards. The reporting duty is moving closer to AI behaviour itself. Most organisations have incident-response plans built around an attacker. An AI agent can cause trouble without one.

• 03 / FAILURE MODE –

The system may be working

The classic security incident has a reassuring shape – unauthorised access, detection, containment. AI incidents are different. The system may be authenticated. Its credentials may be valid. The data may be available. Every individual action looks ordinary when separated from sequence.

Examples: expense-processing agent misreads instruction, recruitment tool excludes applicants via proxy variable, customer-service agent discloses cross-account info, research agent uses exposed credential. None requires an attacker.

The OECD defines an AI incident broadly as an event or series of events in which an AI system directly or indirectly causes harm. A cyber investigation asks who compromised the system. An AI investigation must also ask what the system did while operating normally. If the organisation cannot recognise an AI incident, the reporting clock becomes irrelevant.

• 04 / REPORTING THRESHOLD –

The clock starts before certainty

Real incidents don’t arrive with a title. First signal may be unusual database query, customer complaint, unexplained system action. Under CERT-In framework, companies don’t need to complete investigation before initial report.

The six-hour period runs from when organisation notices or is informed. The company needs enough visibility to recognise the event, enough authority to contain it and enough discipline to report what it knows. That is not principally a technical capability. It is an operating model.

• 05 / DECISION RIGHTS –

Your steering committee cannot help

A steering committee can approve policy, review risk and ask for another dashboard. It cannot revoke a credential at 2.17am. The committee owns the meeting, rarely the decisions.

Decision-Rights Assignment

Clear pre-authorised human responders are required to meet tight regulatory windows.

• 06 / SYSTEM OWNERSHIP –

An agent needs a manager

An AI agent has an objective, access and freedom to decide how to use that access. It may call tools, search systems, move data, generate code, alter records or contact people.

A named owner should know: what the agent is expected to do, which systems/tools/datasets it can access, which credentials it uses, what actions require human approval, which behaviours trigger suspension, who can revoke access, what evidence is recorded, which vendor supports the system, which regulators may need notification. An organisation that cannot produce this information quickly does not control the agent. It merely hosts it.

“An organisation that cannot produce this information quickly does not control the agent. It merely hosts it.”

• 07 / EVIDENCE –

Build the evidence before the incident

When AI goes off-script, organisations often discover that the evidence they need was never collected. The prompt is available, but the tool calls are not. The vendor keeps logs, but only for seven days. The evidence bundle below outlines what to preserve.

• 08 / RESPONSE PROTOCOL –

The first six hours

A six-hour playbook should produce controlled action under uncertainty.

• 09 / VENDOR DUTIES –

Your vendor is part of the incident

Many organisations don’t operate the model they depend on. When the system fails, the incident plan may depend on evidence held by a vendor with four-day response target.

Contract should establish: what constitutes vendor-notifiable AI incident, notification speed, logs preserved, model/version information access, subcontractor incident handling, joint investigation leadership, remote suspension capability, model update effects, regulatory reporting cooperation, evidence accessibility after termination.

“A vendor cannot own your regulatory accountability. It can make that accountability easier or almost impossible to discharge.”

• 10 / SHADOW SYSTEMS –

Shadow AI becomes evidence debt

An organisation cannot report an incident involving a system it doesn’t know exists. Every unofficial assistant, browser extension, automated workflow creates unrecorded authority. Missing information becomes evidence debt. AI register should record actions as well as applications. The unit of governance is not the product. It is the authority granted to it.

• 11 / SIMULATION –

Run the drill before the agent does

Use a plausible scenario – finance agent finds credentials in old document, enters procurement system outside approved workflow, downloads supplier records.

Now ask: Who receives alert? Does person recognise AI incident? Who can pause agent? Can organisation preserve decision chain? Who determines if reportable? When does six-hour clock begin? Does it involve personal data? Who contacts vendor? Who informs business owners? What happens to other agents with same permissions? Who decides when system restarts? Measure time, not answers. That delay is the real finding.

• 12 / THE GOVERNANCE TEST –

Governance is the ability to interrupt

Responsible-AI programmes have produced principles – Trust, Fairness, Transparency, Human oversight, Accountability. An incident turns nouns into verbs. Can someone inspect the system? Stop it? Establish what it did? Support affected people? Tell the regulator before deadline? Decide if safe to restart?

If not, the organisation has governance language, not AI governance. India’s six-hour requirement makes the gap visible but the problem is global. AI systems gaining ability to act at machine speed inside organisations that assign responsibility at meeting speed.Every agent needs an owner with the authority to intervene. Incidents require evidence, while each reporting duty needs a rehearsed route from detection to decision. The AI agent may go off-script. Your organisation cannot afford to improvise the response.

Sources and methodology

Key references used in this analysis

This article draws on incident reporting frameworks, vendor accountability standards and AI governance research. The references below support the operational recommendations and regulatory context presented throughout the piece.