How brands can remain visible in the AI age

The commercial imperative is growing quickly: McKinsey has estimated that, by 2028, $750 billion of search revenue worldwide could flow through AI-based search, equivalent to 75% of the approximately $1 trillion search market today.

Instead of reviewing websites, buyers are trusting AI-generated recommendations and shortlists that increasingly direct their attention, preference and spend. A customer exploring a jewellery purchase, a new pair of trainers, a festival outfit, or a product offered by an online retailer may be presented with a tightly curated set of options before ever engaging with a brand directly.

Brands risk lost sales if they are not included within these gen-AI recommendations, but this requires them to change their approach to the blocking of web crawlers and take further steps to boost the visibility of their products while controlling the information that displays in a way that best preserves the reputation of their brand, their customer relationships and their intellectual property (IP) rights.

Changing consumer habits and technology

Gen-AI tools are quickly becoming trusted shopping advisers. Recent research suggests that 58% of consumers already use AI for product or service recommendations, and in fashion adoption is even higher. 

Data from a February 2026 survey by Bain showed that 53% of UK consumers always or mostly use AI when shopping for fashion. This rises to 61% in the US. Adoption spans all generations, with millennials leading at 51%, followed by ‘Gen Z’ (42%), Gen X (36%) and ‘baby boomers’ (32%).  

At the same time, the technology itself is evolving quickly. The latest development is agentic AI: systems that act on the user’s behalf. In the e-commerce context, these agents are active participants in the purchasing journey. Agent-based systems operate like independent assistants, selecting and combining actions to achieve a goal. Their influence now spans the entire customer lifecycle – from surfacing information on products in the form of tailored recommendations and making comparisons between products – including on price – to enabling virtual try-ons and completing transactions through emerging agent-led payment protocols.

As AI-driven discovery evolves at pace, brands that do not adapt risk being excluded from the conversations they no longer control. 

Invisible by design 

Currently, many brands limit their visibility to AI by blocking web crawlers – software that scans and scrapes data from websites – to prevent brand misuse and protect their content, control their image and voice, and safeguard their customers. 

Website product descriptions, campaign imagery and brand narratives are valuable assets protected by a variety of IP rights. Preventing automated copying reduces instances of misuse without permission. 

Brands across the retail and consumer sector carefully manage how they are seen, from imagery and tone of voice to product descriptions, pricing, availability and customer service propositions. Blocking scraping helps avoid uncontrolled reuse or distortion of content. Scraped content can also be used to create convincing fake websites or listings. Restricting access helps make this harder and so proactively defends against counterfeits, fraud and brand misuse, safeguarding customers and their data. 

In a world where the website sat at the heart of the customer journey, these protections came with limited downside. For businesses built on brand trust, distinctive positioning, product quality and availability, customer services and, in some cases, exclusivity, restricting automated access to their website content has been a natural extension of how product distribution, brand presentation and customer relationships are managed. 

For some retailers, there is an additional concern that AI-enabled search and shopping agents may make price comparisons easier, surface cheaper alternatives or divert customers to marketplaces and unofficial sellers. However, complete invisibility may be a greater commercial risk if it means the brand is excluded from the AI-generated shortlist altogether.

In the age where there is growing use of AI tools for online search, discoverability drives demand. Data collected by ‘friendly’ crawlers is what allows AI-powered systems to index, interpret and surface content in response to customer queries. Blocking them may reduce brand misuse, but it also results in invisibility at the very moment customers are deciding what to buy.

Blocking web crawlers also comes with other risks: AI systems may rely on third-party sources, outdated descriptions or even incorrect information, with the risk that brands are represented inaccurately to customers, undermining its intended purpose. For example, it could mean that important product positioning points featured on a brand’s website are missed or simplified. There is also a risk of unofficial sellers, dupes, counterfeits or other poor-quality listings being surfaced by AI systems in the absence of authoritative content from a brand.

Ensuring AI discoverability

The answer for retail brands is not simply to stop blocking crawlers.

AI systems rely heavily on external, authoritative signals when generating answers. Strengthening the quality and consistency of those signals should therefore be a priority. 

Retail and consumer brands should ensure that information about them is accurate and aligned across trusted sources. Editorial coverage should reinforce the correct positioning, while product attributes, sizing, availability, sustainability credentials, service propositions and provenance and brand stories should be well documented and consistently reflected. Recent credible reviews and positive mentions across the internet also play an important role in reinforcing AI confidence and increasing visibility. 

Brands should also refine their approach to data sharing. Controlled engagement with ‘friendly’ platform agents, including licensing models or similarly structured access, can help balance visibility with protection, while ensuring customer traffic is directed back to owned channels. 

In parallel, brands should upgrade their approach to IP protection and brand governance in the context of AI. This includes actively monitoring how the brand appears in AI-generated outputs, testing visibility regularly and addressing inaccuracies where possible. The focus should shift from simply restricting access to managing how the brand is represented and surfaced. 

Optimising owned channels is also important: even if a brand welcomes crawlers on its website, details of its product will not be surfaced in AI recommendations if the AI system cannot read the website. Content needs to be structured in a way that enables AI models to easily read, synthesise and reference it. Product descriptions should be complete, consistent and rich in detail, with clearly stated attributes such as materials, dimensions and provenance. Information should be aligned across all owned channels, mirroring how a skilled in-store retail assistant would present a product.  

At the same time, brands should look to strengthen their channels by investing in AI-enabled experiences. This includes developing conversational agents on web and app platforms to deliver more personalised, and useful interactions, from discovery and inspiration through to purchase and post-purchase support. Creating incentives for clients to re-engage directly with the brand is also important: the focus should be on enhancing, not replacing, the direct customer relationship. 

Brands can make their website easily readable by large language models (LLM) while also curating sites that meet customer expectations and incorporate strong visual merchandising, rich content and engaging experiences, which can otherwise be the enemies of LLM readability and pose a risk of brand distortion. They can do this by operating a compelling customer facing website in tandem with a more basic subsite that is invisible to customers but that satisfies the needs of LLMs. An LLM-friendly subsite may increase the likelihood that AI systems draw on authoritative brand data, improving the prospects of inclusion in relevant recommendations and more accurate brand representation. 

Investment in upgraded AI tools and expertise – with training to ensure employee literacy – is also essential, and it is likely that brands will need to alter some internal processes to adapt to these changes in consumer journeys and technology.

Five takeaways for brands 

• Blocking crawlers can have direct commercial consequences in terms of how customers discover a brand, even where the reason for blocking is to reduce misuse, limit unfair price comparisons or protect customer relationships;
• The customer journey is shifting away from websites. Decisions are increasingly influenced before a user ever visits them; 
• Visibility now depends on machine-readable information. If systems cannot access or understand content, they may ignore it; 
• Providing the right crawler access to the right sources may offer better protection for brands than blocking everything; 
• This is a strategic, business change and not a technical upgrade. It affects brand visibility, positioning and revenue and requires board-level attention. 

Co-written by Gill Dennis of Pinsent Masons.