Financial services firms need to secure oversight of suppliers’ AI use

In a new report that explores the implications for financial services firms of suppliers’ use of AI tools to deliver critical services, Pinsent Masons cited UK financial services regulations that require financial services firms to have oversight of the services they receive from third parties.

The report highlighted how firms need to give thought to how they can exercise effective oversight outside of a formal audit exercise, the rights to which should be secured and would operate separately. It said the incorporation of AI into the way suppliers deliver services has implications for oversight, audit and governance provisions in the contracts firms operate with those suppliers.

“Where AI is part of the supplier’s toolset rather than the subject of the contract, the focus of the customer’s oversight will be on assurance of the service behaviour and outputs/outcomes, risk management, and transparency of AI-driven processes, rather than assessment of the AI product itself,” said Mhairi Mival of Pinsent Masons.

“The customer does not need to assess the supplier’s underlying models as if they were procuring an AI system; however, they do need confidence that AI use does not undermine service quality, controls, resilience, regulatory compliance or customer outcomes,” Mival added.

According to Angus McFadyen, also of Pinsent Masons, firms will also want visibility into how AI contributes to certain supplier processes – such as its automated triaging, monitoring, optimisation and quality assurance – as well as assurances that controls exist to prevent harmful behaviours such as biased outputs, automation errors or systemic failures. Firms do not need to engage in a “technical deep dive” to understand how such models work as they are changing constantly; they should instead focus on “guardrails and evidence based assurance of behaviour, accuracy, error handling and exception management, as these relate to the services – critically, as AI is used in a more agentic manner, the focus needs to extend beyond the output to the behaviour”, McFadyen said.

McFadyen said financial services firms should consider requiring suppliers to disclose documents that record how AI models are managed and validated, as well as validation summaries, or drift/monitoring results, as opposed to the model code or full datasets. These could be supplemented by rights to inspect how the supplier supervises, tests and updates AI tooling, as well as transparency solutions, he said.

“Suppliers typically resist disclosure of proprietary algorithms, and customers often do not need these details at a level that presents risk to the supplier’s intellectual property rights – what matters is safety, security, transparency, and accountability,” McFadyen said. “This includes documentation showing how AI supports service delivery and evidence of relevant controls.”

The report highlighted how financial services firms may not always have “direct visibility” of suppliers’ AI use. Mival said it is therefore important for firms to negotiate contract provisions that require suppliers to notify them “when and how integrated AI is used to deliver services and notify firms in relation to any issues relating to the use of AI that may have an impact on the service”.

“These should cover material AI model failures, instances where AI-generated outputs are found to have been incorrect or unreliable after being incorporated into deliverables, suspected exposure of customer data, and any change to the supplier’s AI tooling or oversight model that could affect the quality or risk profile of the services,” Mival said. “Notification timelines should reflect the customer’s own operational resilience reporting obligations and the expectation that regulators may require the customer to account for AI-related incidents in outsourced functions. The supplier should also be required to cooperate with any regulatory inquiry relating to AI-enabled services delivered to the customer, including by providing access to relevant documentation and personnel.”