Supplier AI use poses data risks for financial services firms

According to a new report published by Pinsent Masons, explicit prohibitions against the use of data for the training of AI models and robust controls over the AI supply chain are essential for the protection of firms’ proprietary interests and for regulatory compliance.

iThe report highlighted how suppliers are increasingly using AI to deliver services to their customers – including financial services firms – and may seek to use aggregated or anonymised data in doing so, for purposes such as benchmarking, product development or service improvement. It flagged risks this raises around the use and protection of customer data that financial services firms must consider, to address the inadvertent processing and disclosure of personally identifiable or commercially sensitive data.

Angus McFadyen and Matthew Godfrey-Faussett of Pinsent Masons said the risks need to be carefully managed through effective contractual provisions and technical architecture and controls.

McFadyen said: “When a supplier processes customer information as part of its service delivery, there can be incentives for that data to be used not merely for the immediate processing task but also as training material to improve related AI models, whether those of the supplier or a third-party provider. From the supplier’s perspective, this represents an opportunity to enhance service quality continuously, since many AI systems rely on large, diverse datasets which access and re-train on additional rare or unique data. From the customer’s perspective, however, this practice can raise fundamental concerns.”

“If a financial services business’ customer data contributes to improving a model that subsequently serves its competitors, the customer has in effect subsidised the development of capabilities that erode its competitive position. More immediately, the use of customer data for training purposes creates data protection issues: where the data concerns identifiable individuals, an appropriate legal basis for such processing may not have been secured, or transparency ensured. For these reasons, the contractual default position should be an explicit prohibition on using customer data for AI model training where it is outside of the core service provision, whether by the supplier or any third-party provider in the supply chain,” McFadyen added.

Godfrey-Faussett said financial services firms should ensure that suppliers require their “explicit agreement” to depart from the default position and further highlighted other contractual terms necessary to protect against data-related risks.

“Customers must ensure that contractual audit rights are broad enough to extend beyond standard information security reviews to include inspections of data pipelines and confirmation that customer data has not been incorporated into the supplier’s or any third party’s AI model – in fact, many regulators require audit to be unrestricted and so this is often in place for material engagements,” Godfrey-Faussett said. “Equally important are provisions addressing data outputs. Where the AI system generates inferences, scores, or profiles based on customer data, the contract should clarify use and control of these derivatives and restrict the supplier’s ability to retain, aggregate, or commercialise them as appropriate.”

“The contract should also require the supplier to only use approved AI tools operating under enterprise agreements that contractually exclude customer data from independent model training and limit retention of prompts and outputs. A blanket prohibition on inputting the institution’s data into consumer-facing AI interfaces should be explicit, not implied,” Godfrey-Faussett added, citing the further need for contracts to require suppliers to confirm “not only that data has been deleted from operational systems” upon exit of arrangements but that the data “has not been incorporated into any model or training dataset that will survive termination of the relationship”.

Access the Pinsent Masons report on AI-enabled service delivery