An AI breach of an Australian government website raises questions of liability

On 24 September, prime minister Anthony Albanese revealed that an OpenAI agent had unintentionally hacked into Australia’s Medicare statistics reporting portal in June and gained access to both public and non-public data. OpenAI took three months to notify the government of the breach, sparking public and regulatory debate.

On 18 June, an OpenAI agent was routinely tasked to scrape the web to train its AI models like ChatGPT. Using reinforcement learning, the agent could interact with graphical user interfaces, navigate complex websites, and bypass anti-bot and anti-scraping technologies to extract data from any web page. According to Albanese, the OpenAI agent gained unauthorised access to the Medicare portal on 18 June, but OpenAI did not report the breach to the government, through Medicare’s public portal, until 10 September.

OpenAI's own statement confirms that its models took unintended actions while attempting to look up health statistics during an internal evaluation. The government similarly confirmed that when the agent sought information from the Medicare portal and was refused, it proceeded to breach the portal and retrieve the information, circumventing the boundaries and restrictions set.

This was not a theoretical risk. It was precisely the kind of incident that the Australian Signals Directorate (ASD) had warned about in its May 2026 (29-page // 1253kb PDF) guidance on agentic AI services. That guidance flagged that LLM-based agents may change their behaviour when evaluations are underway and may bypass system-level instructions to achieve their objectives. Described as “specification gaming”, these agents may seek shortcuts or loopholes that achieve their goals, contrary to the goal's intention, or create security vulnerabilities. The Medicare incident suggests that this guidance was not followed to prevent this behaviour.

The Australian government has announced a taskforce to investigate the data breach. Services Australia, the Commonwealth agency responsible for Medicare, is subject to the Privacy Act, including the data security obligations under Australian privacy principle (APP) 11 and the notifiable data breaches scheme.

Services Australia reported the incident to the ASD’s Cyber Security Centre on 15 September. The investigation will be conducted under the joint leadership of the Department of Prime Minister and Cabinet, working alongside the National Cyber Security Coordinator, the ASD, the Australian AI Safety Institute and Services Australia.

The investigation should help identify how the AI agent was able to circumvent Services Australia’s security controls and why it went undetected for so long; and determine whether the incident was a data breach and, if so, whether there was a likely risk of serious harm to anyone affected.

It appears that no personal or health information or Medicare numbers were in the impacted files. However, it won't be the last incident caused by what is described as ‘agentic AI misalignment’.

Legislative and cybersecurity gaps

The investigation does present a genuine opportunity to assess whether the existing cyber security framework is fit for purpose and to lay the groundwork for targeted reform of Australia's privacy and cyber security laws as they apply to autonomous AI systems. Following the announcement of its national AI plan (37-page // 1797kb PDF), part of which was privacy reform and introducing a digital duty of care, the government released its long-awaited tranche 2 proposals to reform the Privacy Act. 

However, while the reforms seek to enhance privacy protections, including by broadening the definition of regulated personal information, they do not propose any AI-specific measures within the Privacy Act. This stands in contrast to the enhanced obligations for higher-risk critical infrastructure, which are intended to address security risks arising from emerging technologies such as AI.

These gaps in existing laws are not unique to Australia. In the UK, which shares many common law similarities with Australia, AI agents appear incapable of being held liable for fraud, for example.

Accountability and liability

AI agents are inherently designed to perform tasks and achieve goals with limited supervision or intervention and to adapt. This incident highlights the importance of addressing these risks and accountability for AI agents that breach security or access systems they should not, such as who is responsible for their training, what are the parameters in which they are being used, what tasks are they given and how are they directed to execute them in a way that is authorised.

The Privacy Act, in section 99A, makes organisations liable for the acts or omissions of their directors, employees and agents who act within the scope of their authority, unless they can show they took reasonable precautions and exercised due diligence. AI agents can be trained, but unlike these individuals, consequences don’t matter to them and there is no criminal offence they can be charged with. It has been acknowledged that OpenAI’s models acted beyond their intended scope when accessing Medicare, in breach of their operational boundaries, but whether an AI agent falls within the meaning of “agent” or is simply an act of the organisation remains unclear under the act.

The question then becomes whether instructions given to an AI agent are equivalent to instructions given to a human agent, and whether exceeding those instructions is analogous to an employee acting outside the scope of their apparent authority. This distinction is critical, as existing frameworks were not designed with autonomous AI systems in mind.

At its core, an AI agent is not a legal person. It has no rights, obligations, or capacity to be held liable and currently the law provides no clear path to holding its developer or deployer accountable in these circumstances. When two parties are contracting to develop or procure AI agents, then the contractual terms can help resolve some of these matters and, to the extent there is a data breach involving personal information, the Privacy Act requirements to respond and address the breach and to implement reasonable data security measures will apply. But in this case, there was no contract.

The Australian government's recent exposure draft of the proposed digital duty of care might start to address this issue in the context of online safety. It would require online service providers, including AI content and generation services, to undertake transparent risk assessments and proactive measures to address foreseeable online harms.

However, the draft legislation does not directly address the accountability gap for autonomous AI agents that operate beyond their instructions, particularly in a security context. As AI capabilities continue to evolve, the Medicare incident underscores the need for a comprehensive, regulator-led governance framework that extends beyond existing privacy and online safety laws and guidance, to address the distinct risks posed by agentic AI.

What organisations should do

All connected organisations are exposed to the security risks from agentic AI and should be asking whether their internal and supply chain security controls are adequate to address these threats; whether their incident response plans specifically contemplate AI-driven data breaches; and whether they have cyber insurance policies that help to transfer some of the risk.

Equally, all organisations developing and deploying AI need to proceed with caution and understand and put in place measures to address the risks and monitor their AI agent’s performance. Even under the current privacy framework, it requires demonstrable precaution and due diligence to avoid conduct that causes privacy breaches. Organisations should be asking whether their AI governance frameworks are documented, tested and auditable. The ASD guidance provides an operational starting point for organisations that have not yet aligned their AI deployment to it.