Human oversight may not be enough when managing AI risks

For example, the EU AI Act requires certain high-risk AI systems to be subject to effective human oversight, while Australia's voluntary AI governance principles emphasise the importance of human review, intervention and accountability. The underlying assumption is if a person remains ‘in the loop’ or supervises, AI errors, biases, flawed recommendations or aberrant behaviour are more likely to be detected before they cause harm.  However, there remains questions of how legitimate this assumption is.

Human oversight undoubtedly can introduce factors into decisions processes that AI either performs poorly or doesn’t factor. Humans can apply ethical and moral frameworks, contextualise and apply judgement. Experienced and AI educated humans can often identify unusual outcomes or recognise when an AI-generated output is mere hallucination.

Unfortunately, comprehensive research indicates that human decision-makers are themselves subject to biases and cognitive limitations that can undermine their ability to effectively supervise AI systems. In some cases, the biggest risk may not be the AI model itself, but the employee responsible for reviewing its outputs.

Supervising humans are susceptible to automation bias, which is the tendency for people to place excessive trust in recommendations generated by automated systems, even when those recommendations are incorrect. When an AI tool produces an answer that appears authoritative, users are less likely to critically evaluate its accuracy.

A related issue is confirmation bias. Employees, like all decision-makers, naturally tend to seek information that confirms their existing beliefs and assumptions. Confirmation bias might influence the types of prompts used which result in AI systems responses inevitably providing outputs reinforcing preconceived ideas.  Ideally prompts should be drafted neutrally without any embedded bias in the request to facilitate unbiased outputs.

Automation or confirmation bias can be particularly problematic in recruitment, performance management, compliance monitoring and other workplace decisions where objectivity is critical.

Another concern is decision complacency. As AI systems demonstrate increasing levels of competence, employees may gradually become less engaged in the decision-making process. Over time, users can shift from active evaluation to passive monitoring to rubber stamping.

This phenomenon has been observed in highly automated environments such as aviation and industrial operations, where operators may struggle to intervene effectively when systems behave unexpectedly. Similar risks are emerging in workplaces that rely heavily on AI-assisted decision-making. If employees routinely defer to AI recommendations, their capacity to identify errors may diminish precisely when intervention is most needed.

These challenges become particularly acute where employees are under time or efficiency pressure, lack robust frameworks accumulated through experience and training to identify erroneous results or a detailed understanding of how AI systems operate. Rather than functioning as an independent check, the human reviewer may simply endorse the AI's recommendation without meaningful scrutiny or without the judgement or experience to identify erroneous outputs. 

The longer-term effects of AI use on human capability also deserve attention. Some researchers have raised concerns about cognitive decline resulting from the incorrect use of AI. While AI tools can improve productivity, inappropriate use of AI may adversely impact our capacity to develop or maintain critical thinking, problem-solving and independent judgement.

If employees become accustomed to delegating tasks to AI, organisations may face an unintended consequence of a workforce that is increasingly less capable of exercising the oversight regulators expect. The skills required to scrutinise AI outputs may weaken over time through lack of use.

These issues have important implications for organisations seeking to comply with AI governance obligations. Simply requiring a human review stage may not be sufficient to demonstrate effective oversight, and regulators are increasingly likely to focus on whether human intervention is meaningful rather than merely procedural.

Organisations must consider how work is designed around AI systems. Organisations should consider the experience and capability of employees using AI and whether their frameworks are sufficient to adequately supervise AI, such as in the case of new employees versus experienced employees. Training should not only cover how to use AI tools, their weaknesses and strengths, but also educate employees about human cognition and weaknesses like automation bias, confirmation bias and other cognitive risks. Staff should understand that AI outputs require critical assessment, even when recommendations appear persuasive or are generated by systems with strong performance records.

Periodic reviews of human supervised AI workloads and potentially rotating in AI supervisors may alleviate complacency and bias incrementally creeping into decisions or outcomes.

Workloads also matter because employees who are expected to review large volumes of AI-generated outputs under significant time constraints are less likely to apply the cognitive effort to spot errors. Indeed, performance metrics which reward task completion, efficiency and productivity may inadvertently lead individuals to focus less on supervision and more on task completion. Effective oversight requires sufficient time, authority and expertise to challenge system recommendations where appropriate.

As AI regulation continues to evolve, human oversight will remain a central element of risk management frameworks. However, policymakers and businesses should be careful not to assume that placing a person in the loop automatically reduces or removes risk.

Human judgement remains essential, but it is not infallible. The effectiveness of AI oversight depends not only on the technology being supervised, but also the capability, awareness and behaviour of the people responsible for supervising it. In that respect, managing human factors may prove just as important as managing the AI itself.