EU guidance helps clarify AI web scraping requirements

The guidelines (23 pages/294KB PDF), adopted for public consultation on 7 July, “do not answer every question, but do provide some important guidance for platform operators”, said Wouter Seinen, cyber, data and technology law expert at Pinsent Masons.

“The guidelines offer greater certainty both for platforms seeking to innovate responsibly and for those seeking to prevent unauthorised AI training on their users' content,” said Seinen.

Web scraping is the automated process of collecting information from websites using software tools rather than a human manually copying and pasting data. The process uses automated tools to extract and store information from publicly available web services, such as public registers, news websites, social media platforms, discussion forums and blogs. The method has become common for gathering large datasets needed to train large language models and other gen-AI systems. However, the regulator warned that large-scale scraping often takes place without individuals’ knowledge and can pose significant risks to both privacy and data protection rights.

Among the recommendations are measures intended to support compliance with the GDPR’s data minimisation principle. The EDPB said organisations should consider whether synthetic data could be used instead of personal data before any collection takes place. Firms should also define precise collection criteria, conduct data mapping and inventory exercises, deploy filters to exclude certain categories of data, and avoid scraping websites that are structurally likely to contain sensitive information or that clearly object to scraping activities.

The guidance also addresses measures that can be taken after data has been collected. According to the EDPB, controllers could use syntax-based filtering to remove unnecessary information and, where feasible, replace real-world data with synthetic datasets or anonymise or pseudonymise personal information before it is used for AI training.

A significant portion of the guidance focuses on the use of the GDPR lawful basis of legitimate interests, which many private sector AI developers have relied upon when collecting publicly available online information. The EDPB reiterates that organisations relying on Article 6(1)(f) GDPR must satisfy the three-part test requiring a legitimate interest, necessity of processing, and a balancing exercise weighing the organisation’s interests against the rights and freedoms of affected individuals.

The regulator said controllers can strengthen their position in that balancing test by implementing safeguards such as excluding particular categories of personal data from collection; limiting scraping to freely accessible information; increasing transparency around scraping activities; facilitating the exercise of individuals’ rights; and deleting, anonymising or pseudonymising personal data as early as possible. The guidance also addresses transparency, acknowledging that individual notices may, depending on the circumstances, be impossible or involve disproportionate effort. In such situations, controllers should consider appropriate alternative ways of informing affected individuals such as publishing statements on their website or other appropriate public channels. Additional measures designed to reduce risks associated with AI models, including memorisation and regurgitation of personal data, are also highlighted in the guidance.

Seinen said: “For platform operators, the guidelines point in two directions. First, they confirm that user expectations matter, and those expectations are influenced by what the platform has told users about how their data may be used. A platform that has been transparent about broader business uses of content is likely to be in a stronger position when assessing AI training activities. Second, the EDPB recognises that platforms can draw meaningful boundaries around their data. If a platform uses technical measures such as ‘robots.txt’, ‘ai.txt’, login restrictions or CAPTCHA protections, and makes clear that it opposes AI training uses, third parties cannot simply point to public accessibility of content as a justification for scraping it.”

The guidelines also tackle one of the most challenging areas for AI developers – the treatment of ‘special category’ data, such as information revealing health conditions, political opinions or religious beliefs. The GDPR generally prohibits processing such information unless a specific exemption applies.

Malcolm Dowden, data protection law expert at Pinsent Masons said: “According to the EDPB, a judgment of the Court of Justice of the EU in GC and others may provide a basis for assessing situations where special category data is collected only incidentally during web scraping. However, the EDPB stresses that controllers must carry out a case-by-case analysis to determine whether the following conditions are met. First, the processing activity must have relevant similarities with the processing activity of a search engine, as referred to in the court's judgment; second, the processing must involve only incidental and residual processing of special categories of personal data, with no intentional collection; third, it must be difficult or impossible to assess whether and to what extent the processing includes special categories of personal data, and therefore to prevent the collection of such data; and fourth, the controller must implement measures 'within the framework of his responsibilities, powers and capabilities' to prevent the collection and dissemination of special categories of personal data”.

“The EDPB emphasises that the controller must be able to demonstrate, in line with the accountability principle, that these conditions are met and that the measures adopted are relevant and effective,” he said.

The draft guidance is now subject to public consultation until 30 October. The EDPB has not yet indicated when a final version will be adopted.