AI Agents Blocked by Websites: The Next Big Hurdle

AI Agents Blocked by Websites: The Next Big Hurdle

The Rise of Personal AI Agents and the Blocking Problem

Personal AI agents, like Meta’s Muse, Instinct, ChatGPT’s Dots, and others, are kicking off a new wave of consumer AI that involves more than just responding to queries. These AI agents can actually get things done on users’ behalf, like booking flights, making dinner reservations, ordering groceries, and more, all without having to be technical enough to set up OpenClaw or some other agent on your own computer.

But consumers adopting these agents are often running into issues when the website on the other end of their request blocks the AI from completing the job. Most recently, Amazon began blocking Meta’s Muse AI agent from its retail site, meaning that the agent could no longer browse or make purchases from its product catalog.

However, the problem is not limited to Amazon or Muse. User complaints across social media indicate their agents are often running into similar blocks. Some of these are intentional, like Amazon’s, while others are the result of websites’ traditional anti-bot measures designed to protect against spam and malicious activity. What’s not clear to the end user is whether or not the block is accidental on the website owner’s part, which leaves them frustrated with both the service provider or retailer and the agent trying to facilitate the transaction.

Why Blocks Happen: Intentional vs. Accidental

Not every block is deliberate. Some are intentional, like Amazon's decision to block Meta's Muse AI agent from browsing or making purchases on its retail site. Others stem from websites' traditional anti-bot measures, which are designed to protect against spam and malicious activity. For users, the trouble is that it isn't clear whether a block is accidental on the website owner's part.

Walmart is a case in point. TechCrunch saw complaints on social media claiming Muse couldn't complete a purchase on Walmart's site, and NBC reported Walmart had faced this problem in its own tests. But a Walmart spokesperson said these blocks weren't intentional, noting the company is partnered with Muse. The issue is that Walmart's site presents a button visitors must click to verify they're human. If that experience is interrupted, the check can fail and the agent gets booted out.

That confusion leaves users frustrated with both the retailer and the agent trying to complete the transaction.

Company Responses and User Complaints

Some companies have drawn clear lines against unauthorized agents. Delta said it is taking steps to protect customers from unauthorized automated activity, and United's Terms of Use prohibit using any robot, spider, or automatic device for unauthorized purposes without prior written permission. Yelp does not permit non-human traffic unless the agent has paid for content and data access through its data licensing program. eBay said its policies restrict unauthorized agents and actions like automated scraping and model training, rather than banning all third-party shopping agents.

Early adopters' complaints across social media referenced brands reportedly blocking AI agents, including Yelp, eBay, Zillow, Pizza Hut, Adidas, and various airlines, with some believing the blocking activity recently increased. A couple of people even said eBay suspended their accounts because of their use of agentic AI. Another complained that Google kicked out Instinct while it was cleaning up their Gmail inbox.

Industry Efforts and the Path Forward

To address the blocking problem, industry partners including Meta, Walmart, Stripe, Sierra, Genesys, Rocket, NiCE, and Decagon have begun working on an open standard that would dictate how AI agents can communicate with businesses, helping to separate the good bots acting on behalf of the user and the bad. This protocol will be specifically focused on agent-to-agent communication for the purposes of online commerce, Meta said in a blog post.

Cloudflare is another factor. A September 15 change to crawler defaults allows site owners to block AI training while allowing other types of bots, and existing sites set to block AI bots for security reasons were shifted to block AI agents on any pages that had ads. Cloudflare also runs a marketplace where AI bots have to pay for the data they access, and recently began testing a new web browser built specifically for AI agents.

Meta’s decision to form specific brand partnerships for Muse access seems to be the right way to go, as is its decision to focus on new standards. Meta also maintains a list of “connectors” inside the Muse app, which continues to grow, and a bigger reorganization of that list is likely soon to be underway.

E-commerce  AI agents 

Comment